Recent work on the Retrieval-Enhanced Transformer (RETRO) model has shown
that off-loading memory from trainable weights to a retrieval database can
significantly improve language modeling and match the performance of
non-retrieval models that are an order of magnitude larger in size. It has been
suggested that at least some of this performance gain is due to non-trivial
generalization based on both model weights and retrieval. In this paper, we try
to better understand the relative contributions of these two components. We
find that the performance gains from retrieval largely originate from
overlapping tokens between the database and the test data, suggesting less
non-trivial generalization than previously assumed. More generally, our results
point to the challenges of evaluating the generalization of retrieval-augmented
language models such as RETRO, as even limited token overlap may significantly
decrease test-time loss. We release our code and model at
this https URL

该研究探讨了 Retrieval-Enhanced Transformer（RETRO）模型的性能以及性能提升的原因，发现检索数据库可显著提高语言建模的效果，但其性能提升主要来自于测试数据与数据库的重叠标记。该研究还指出，检索增强的语言模型对于其泛化能力的评估存在一定挑战，即使是有限的标记重叠也可能显著降低测试损失。

关于检索增强的 Transformer 的泛化能力

On the Generalization Ability of Retrieval-Enhanced Transformers

We enhance auto-regressive language models by conditioning on document chunks
retrieved from a large corpus, based on local similarity with preceding tokens.
With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO)
obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite
using 25$\times$ fewer parameters. After fine-tuning, RETRO performance
translates to downstream knowledge-intensive tasks such as question answering.
RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked
cross-attention mechanism to predict tokens based on an order of magnitude more
data than what is typically consumed during training. We typically train RETRO
from scratch, yet can also rapidly RETROfit pre-trained transformers with
retrieval and still achieve good performance. Our work opens up new avenues for
improving language models through explicit memory at unprecedented scale.

通过从大型语料库中检索与前面 token 相似的文档块来改善自回归语言模型的条件，并创建 Retrieval-Enhanced Transformer（RETRO），该模型在 Pile 数据集上的表现与 GPT-3 和 Jurassic-1 相当。 RETRO 结合一个冻结的 Bert 检索器，一个可微分的编码器和一个分块交叉关注机制，可以基于比通常在训练期间消耗的数据量高一个数量级的数据预测 token，并在 fine-tuning 后转化为下游的 knowledge-intensive 任务，例如问答等，这一研究为利用显式记忆来提高语言模型的性能开辟了新的途径。