In this paper we improve the zero-shot generalization ability of language
models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves
augmentation documents from multiple information corpora ("external memories"),
with the option to "plug in" new memory at inference time. We develop a joint
learning mechanism that trains the augmentation component with latent labels
derived from the end retrieval task, paired with hard negatives from the memory
mixture. We instantiate the model in a zero-shot dense retrieval setting by
augmenting a strong T5-based retriever with MoMA. Our model, MoMA, obtains
strong zero-shot retrieval accuracy on the eighteen tasks included in the
standard BEIR benchmark. It outperforms systems that seek generalization from
increased model parameters and computation steps. Our analysis further
illustrates the necessity of augmenting with mixture-of-memory for robust
generalization, the benefits of augmentation learning, and how MoMA utilizes
the plug-in memory at inference time without changing its parameters. We plan
to open source our code.

本文提出了一种通过从多个信息语料库（“外部记忆”）检索增强文档的机制，即 Mixture-Of-Memory Augmentation（MoMA），来改善语言模型的零样本泛化能力，并以一种零样本密集检索环境为例，使用 MoMA 增强强 T5-based retriever 模型。MoMA 在标准 BEIR 基准测试中获得了强的零样本检索准确性，优于寻求从增加模型参数和计算步骤进行泛化的系统，并证明了 Mixture-Of-Memory 的必要性及其学习的好处，同时展示了 MoMA 如何利用插入式存储器在推理时不改变其参数。

利用插件记忆混合增强零样本密集检索器

Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories

Lexically constrained decoding for machine translation has shown to be
beneficial in previous studies. Unfortunately, constraints provided by users
may contain mistakes in real-world situations. It is still an open question
that how to manipulate these noisy constraints in such practical scenarios. We
present a novel framework that treats constraints as external memories. In this
soft manner, a mistaken constraint can be corrected. Experiments demonstrate
that our approach can achieve substantial BLEU gains in handling noisy
constraints. These results motivate us to apply the proposed approach on a new
scenario where constraints are generated without the help of users. Experiments
show that our approach can indeed improve the translation quality with the
automatically generated constraints.

本文提出了一种新的框架，将用户提供的限制条件作为外部记忆存储，并以柔性方式进行处理，从而解决在现实世界中限制条件可能存在的问题，实验结果表明，我们的方法能够在处理嘈杂的限制条件时取得相当大的 BLEU 收益，这些结果使我们能够将所提出的方法应用于不需要用户帮助生成限制条件的新场景中并证明了该方法确实能够提高自动生成的约束下的翻译质量。