Recent work on the Retrieval-Enhanced Transformer (RETRO) model has shown
that off-loading memory from trainable weights to a retrieval database can
significantly improve language modeling and match the performance of
non-retrieval models that are an order of magnitude larger in size. It has been
suggested that at least some of this performance gain is due to non-trivial
generalization based on both model weights and retrieval. In this paper, we try
to better understand the relative contributions of these two components. We
find that the performance gains from retrieval largely originate from
overlapping tokens between the database and the test data, suggesting less
non-trivial generalization than previously assumed. More generally, our results
point to the challenges of evaluating the generalization of retrieval-augmented
language models such as RETRO, as even limited token overlap may significantly
decrease test-time loss. We release our code and model at
this https URL

该研究探讨了 Retrieval-Enhanced Transformer（RETRO）模型的性能以及性能提升的原因，发现检索数据库可显著提高语言建模的效果，但其性能提升主要来自于测试数据与数据库的重叠标记。该研究还指出，检索增强的语言模型对于其泛化能力的评估存在一定挑战，即使是有限的标记重叠也可能显著降低测试损失。