Most existing retrieval-augmented language models (LMs) for question
answering assume all retrieved information is factually correct. In this work,
we study a more realistic scenario in which retrieved documents may contain
misinformation, causing conflicts among them. We observe that the existing
models are highly brittle to such information in both fine-tuning and
in-context few-shot learning settings. We propose approaches to make
retrieval-augmented LMs robust to misinformation by explicitly fine-tuning a
discriminator or prompting to elicit discrimination capability in GPT-3. Our
empirical results on open-domain question answering show that these approaches
significantly improve LMs' robustness to knowledge conflicts. We also provide
our findings on interleaving the fine-tuned model's decision with the
in-context learning process, paving a new path to leverage the best of both
worlds.

本文研究了现有检索增强语言模型在面对包含错误信息的检索文档时的健壮性问题，并通过在微调和上下文几次学习中显式微调鉴别器或提示来提高检索增强语言模型对错误信息的容忍度，并揭示了有关在上下文学习过程中将微调模型决策与合并的最佳路径。

分辨和解答：通过鉴别器缓解检索增强模型中错误信息的影响

Discern and Answer: Mitigating the Impact of Misinformation in  Retrieval-Augmented Models with Discriminators

Retrieval-augmented language models such as Fusion-in-Decoder are powerful,
setting the state of the art on a variety of knowledge-intensive tasks.
However, they are also expensive, due to the need to encode a large number of
retrieved passages. Some work avoids this cost by pre-encoding a text corpus
into a memory and retrieving dense representations directly. However,
pre-encoding memory incurs a severe quality penalty as the memory
representations are not conditioned on the current input. We propose LUMEN, a
hybrid between these two extremes, pre-computing the majority of the retrieval
representation and completing the encoding on the fly using a live encoder that
is conditioned on the question and fine-tuned for the task. We show that LUMEN
significantly outperforms pure memory on multiple question-answering tasks
while being much cheaper than FiD, and outperforms both for any given compute
budget. Moreover, the advantage of LUMEN over FiD increases with model size.

LUMEN 是一种检索 - 增强型语言模型，介于传统的基于文本语料库的预编码记忆和需要编码大量检索段落的 Fusion-in-Decoder 之间，利用一个实时编码器完成编码任务，比 FiD 更便宜且更适合于解决各种问题 - 回答任务，而且 LUMEN 的优势随着模型大小的增加而增加。

预计算内存还是即兴编码？一种混合检索增强方法最大化利用您的计算资源

Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute

Retrieval-augmented language models (LMs) use non-parametric memory to
substantially outperform their non-retrieval counterparts on perplexity-based
evaluations, but it is an open question whether they achieve similar gains in
few- and zero-shot end-task accuracy. We extensively study one such model, the
k-nearest neighbor LM (kNN-LM), showing that the gains marginally transfer. The
main challenge is to achieve coverage of the verbalizer tokens that define the
different end-task class labels. To address this challenge, we also introduce
kNN-Prompt, a simple and effective kNN-LM with automatically expanded fuzzy
verbalizers (e.g. to expand terrible to also include silly and other
task-specific synonyms for sentiment classification). Across nine diverse
end-tasks, using kNN-Prompt with GPT-2 large yields significant performance
boosts over strong zero-shot baselines (13.4% absolute improvement over the
base LM on average). We also show that other advantages of non-parametric
augmentation hold for end tasks; kNN-Prompt is effective for domain adaptation
with no further training, and gains increase with the size of the retrieval
model.

研究了 k-nearest neighbor LM 以及 kNN-Prompt 模型在少数样本和零样本任务准确度上的性能，发现其对于扩展任务特定的模糊单词十分有效，且具有在领域适应中快速提高准确度的优势。