Semiparametric language models (LMs) have shown promise in continuously
learning from new text data by combining a parameterized neural LM with a
growable non-parametric memory for memorizing new content. However,
conventional semiparametric LMs will finally become prohibitive for computing
and storing if they are applied to continual learning over streaming data,
because the non-parametric memory grows linearly with the amount of data they
learn from over time. To address the issue of scalability, we present a simple
and intuitive approach called Selective Memorization (SeMem), which only
memorizes difficult samples that the model is likely to struggle with. We
demonstrate that SeMem improves the scalability of semiparametric LMs for
continual learning over streaming data in two ways: (1) data-wise scalability:
as the model becomes stronger through continual learning, it will encounter
fewer difficult cases that need to be memorized, causing the growth of the
non-parametric memory to slow down over time rather than growing at a linear
rate with the size of training data; (2) model-wise scalability: SeMem allows a
larger model to memorize fewer samples than its smaller counterpart because it
is rarer for a larger model to encounter incomprehensible cases, resulting in a
non-parametric memory that does not scale linearly with model size. We conduct
extensive experiments in language modeling and downstream tasks to test SeMem's
results, showing SeMem enables a semiparametric LM to be a scalable continual
learner with little forgetting.

本文提出了一种名为 Selective Memorization 的方法来提高半参数语言模型在不断学习流数据时的可伸缩性，该方法只记忆模型难以处理的样本。实验表明，Selective Memorization 能够实现半参数语言模型的可扩展、持续学习，并减少遗忘。