With direct access to human-written reference as memory, retrieval-augmented
generation has achieved much progress in a wide range of text generation tasks.
Since better memory would typically prompt better generation~(we define this as
primal problem), previous works mainly focus on how to retrieve better memory.
However, one fundamental limitation exists for current literature: the memory
is retrieved from a fixed corpus and is bounded by the quality of the corpus.
Due to the finite retrieval space, bounded memory would greatly limit the
potential of the memory-augmented generation model. In this paper, by exploring
the duality of the primal problem: better generation also prompts better
memory, we propose a framework called Selfmem, which iteratively adopts a
retrieval-augmented generator itself to generate an unbounded memory pool and
uses a memory selector to pick one generated memory for the next generation
round. By combining the primal and dual problem, a retrieval-augmented
generation model could lift itself up with its own output in the infinite
generation space. To verify our framework, we conduct extensive experiments
across various text generation scenarios including neural machine translation,
abstractive summarization and dialogue generation over seven datasets and
achieve state-of-the-art results in JRC-Acquis(four directions), XSum(50.3
ROUGE-1) and BigPatent(62.9 ROUGE-1).

本文提出了一个名为 Selfmem 的框架，结合检索增强生成模型的原始和对偶问题，采用检索增强生成器自身进行迭代，以生成无限的记忆池，并使用记忆选择器来挑选下一代回合的记忆。在多个文本生成场景中进行了广泛的实验，并取得了 JRC-Acquis（四个方向），XSum（50.3 ROUGE-1）和 BigPatent（62.9 ROUGE-1）的最新成果。