Contrast consistency, the ability of a model to make consistently correct
predictions in the presence of perturbations, is an essential aspect in NLP.
While studied in tasks such as sentiment analysis and reading comprehension, it
remains unexplored in open-domain question answering (OpenQA) due to the
difficulty of collecting perturbed questions that satisfy factuality
requirements. In this work, we collect minimally edited questions as
challenging contrast sets to evaluate OpenQA models. Our collection approach
combines both human annotation and large language model generation. We find
that the widely used dense passage retriever (DPR) performs poorly on our
contrast sets, despite fitting the training set well and performing
competitively on standard test sets. To address this issue, we introduce a
simple and effective query-side contrastive loss with the aid of data
augmentation to improve DPR training. Our experiments on the contrast sets
demonstrate that DPR's contrast consistency is improved without sacrificing its
accuracy on the standard test sets.

本文研究了在自然语言处理中，模型在面对扰动时进行一致性预测的能力，提出了一种基于人工标注和大型语言模型生成的问题集，以改善现有模型在开放域问答任务上的表现，对广泛使用的密集式段落检索器模型（DPR）进行了改进，并通过对比性损失和数据增强技术提高了 DPR 的对比一致性，而不影响其在标准测试集上的准确度。

探究最小编辑问句下的开放领域问答系统对比一致性

Exploring Contrast Consistency of Open-Domain Question Answering Systems  on Minimally Edited Questions

We propose Generation-Augmented Retrieval (GAR) for answering open-domain
questions, which augments a query through text generation of heuristically
discovered relevant contexts without external resources as supervision. We
demonstrate that the generated contexts substantially enrich the semantics of
the queries and GAR with sparse representations (BM25) achieves comparable or
better performance than state-of-the-art dense retrieval methods such as DPR.
We show that generating diverse contexts for a query is beneficial as fusing
their results consistently yields better retrieval accuracy. Moreover, as
sparse and dense representations are often complementary, GAR can be easily
combined with DPR to achieve even better performance. GAR achieves
state-of-the-art performance on Natural Questions and TriviaQA datasets under
the extractive QA setup when equipped with an extractive reader, and
consistently outperforms other retrieval methods when the same generative
reader is used.

本文提出了一种 Generation-Augmented Retrieval 方法来回答开放领域的问题，这种方法通过发现相关语境并进行文本生成来增强查询，同时证明所生成的语境大大丰富了查询的语义，使用 BM25 稀疏表示与 DPU 等最先进的稠密检索方法相比，GAR 能够获得同等甚至更好的检索效果。本文还将 G.A.R. 与 DPU 相结合，能够在 Natural Questions 和 TriviaQA 数据集上显着提高提取式 QA 基线的性能，并且在使用相同的生成性阅读器时，GAR 频繁地优于其他检索算法。