Question answering-based summarization evaluation metrics must automatically
determine whether the QA model's prediction is correct or not, a task known as
answer verification. In this work, we benchmark the lexical answer verification
methods which have been used by current QA-based metrics as well as two more
sophisticated text comparison methods, BERTScore and LERC. We find that LERC
out-performs the other methods in some settings while remaining statistically
indistinguishable from lexical overlap in others. However, our experiments
reveal that improved verification performance does not necessarily translate to
overall QA-based metric quality: In some scenarios, using a worse verification
method -- or using none at all -- has comparable performance to using the best
verification method, a result that we attribute to properties of the datasets.

本文研究基于问答的摘要评估方法，要求自动决定 QA 模型的预测是否正确，即答案验证。通过比较词汇重叠和两种更复杂的文本比较方法，BERTScore 和 LERC，我们发现在某些场景下 LERC 的效果优于其他方法，但在其他场景下表现与词汇重叠相当。然而，我们的实验表明，改进的验证性能不一定能够转化为整体 QA-based 评估质量，在某些情况下，使用较差的验证方法（或根本不使用）具有与使用最佳验证方法相当的性能，这是由于数据集的特性所致。

基于问题回答的摘要评估度量的答案验证方法基准测试

Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics

Recent work has shown that an answer verification step introduced in
Transformer-based answer selection models can significantly improve the state
of the art in Question Answering. This step is performed by aggregating the
embeddings of top $k$ answer candidates to support the verification of a target
answer. Although the approach is intuitive and sound still shows two
limitations: (i) the supporting candidates are ranked only according to the
relevancy with the question and not with the answer, and (ii) the support
provided by the other answer candidates is suboptimal as these are retrieved
independently of the target answer. In this paper, we address both drawbacks by
proposing (i) a double reranking model, which, for each target answer, selects
the best support; and (ii) a second neural retrieval stage designed to encode
question and answer pair as the query, which finds more specific verification
information. The results on three well-known datasets for AS2 show consistent
and significant improvement of the state of the art.

本研究提出了一种双重再排序模型和第二个神经检索阶段，以改进在问题应答中使用聚合策略的转换器模型的限制。对三个知名数据集进行的实验表明，该方法在 AS2 上具有相当和显著的改进。

准确问答的双重检索与排名

Double Retrieval and Ranking for Accurate Question Answering

Machine reading comprehension (MRC) on real web data usually requires the
machine to answer a question by analyzing multiple passages retrieved by search
engine. Compared with MRC on a single passage, multi-passage MRC is more
challenging, since we are likely to get multiple confusing answer candidates
from different passages. To address this problem, we propose an end-to-end
neural model that enables those answer candidates from different passages to
verify each other based on their content representations. Specifically, we
jointly train three modules that can predict the final answer based on three
factors: the answer boundary, the answer content and the cross-passage answer
verification. The experimental results show that our method outperforms the
baseline by a large margin and achieves the state-of-the-art performance on the
English MS-MARCO dataset and the Chinese DuReader dataset, both of which are
designed for MRC in real-world settings.

提出了一种基于端到端神经模型的多文本阅读理解模式，通过多个模块的共同作用来检测文本间的答案验证，实现对真实网络数据的阅读理解，显著提高了 MS-MARCO 数据集和中文 DuReader 数据集的 MRC 模型性能。