The performance of Open-Domain Question Answering (ODQA) retrieval systems
can exhibit sub-optimal behavior, providing text excerpts with varying degrees
of irrelevance. Unfortunately, many existing ODQA datasets lack examples
specifically targeting the identification of irrelevant text excerpts. Previous
attempts to address this gap have relied on a simplistic approach of pairing
questions with random text excerpts. This paper aims to investigate the
effectiveness of models trained using this randomized strategy, uncovering an
important limitation in their ability to generalize to irrelevant text excerpts
with high semantic overlap. As a result, we observed a substantial decrease in
predictive accuracy, from 98% to 1%. To address this limitation, we discovered
an efficient approach for training models to recognize such excerpts. By
leveraging unanswerable pairs from the SQuAD 2.0 dataset, our models achieve a
nearly perfect (~100%) accuracy when confronted with these challenging text
excerpts.

本文研究了利用随机策略训练模型存在识别高语义重叠无关文本摘录能力不足的重要局限性，并发现了一种有效的方法来训练模型以识别此类摘录，通过利用 SQuAD 2.0 数据集的不可回答配对，我们的模型在面对这些具有挑战性的文本摘录时实现了近乎完美的准确性（约 100%）。

在检索增强开放域问答中的可回答性

Answerability in Retrieval-Augmented Open-Domain Question Answering

In this paper, we introduce an important yet relatively unexplored NLP task
called Multi-Narrative Semantic Overlap (MNSO), which entails generating a
Semantic Overlap of multiple alternate narratives. As no benchmark dataset is
readily available for this task, we created one by crawling 2,925 narrative
pairs from the web and then, went through the tedious process of manually
creating 411 different ground-truth semantic overlaps by engaging human
annotators. As a way to evaluate this novel task, we first conducted a
systematic study by borrowing the popular ROUGE metric from text-summarization
literature and discovered that ROUGE is not suitable for our task.
Subsequently, we conducted further human annotations/validations to create 200
document-level and 1,518 sentence-level ground-truth labels which helped us
formulate a new precision-recall style evaluation metric, called SEM-F1
(semantic F1). Experimental results show that the proposed SEM-F1 metric yields
higher correlation with human judgement as well as higher inter-rater-agreement
compared to ROUGE metric.

本文介绍了一个重要但相对未被探索的 NLP 任务，称为多叙事语义重叠（MNSO），它涉及生成多个替代叙事的语义重叠。我们使用在网上爬取的 2,925 个叙事对创建了一个基准数据集，并通过人工注释创建了 411 个不同的地面真实语义重叠，提出了一个新的精确度 - 召回率样式的评估指标 SEM-F1。实验结果表明，该指标与人类判断的相关性更高，而且相较于 ROUGE 指标，具有更高的人际一致性。