Our world is open-ended, non-stationary, and constantly evolving; thus what
we talk about and how we talk about it change over time. This inherent dynamic
nature of language contrasts with the current static language modelling
paradigm, which trains and evaluates models on utterances from overlapping time
periods. Despite impressive recent progress, we demonstrate that Transformer-XL
language models perform worse in the realistic setup of predicting future
utterances from beyond their training period, and that model performance
becomes increasingly worse with time. We find that, while increasing model size
alone -- a key driver behind recent progress -- does not solve this problem,
having models that continually update their knowledge with new information can
indeed mitigate this performance degradation over time. Hence, given the
compilation of ever-larger language modelling datasets, combined with the
growing list of language-model-based NLP applications that require up-to-date
factual knowledge about the world, we argue that now is the right time to
rethink the static way in which we currently train and evaluate our language
models, and develop adaptive language models that can remain up-to-date with
respect to our ever-changing and non-stationary world. We publicly release our
dynamic, streaming language modelling benchmarks for WMT and arXiv to
facilitate language model evaluation that takes temporal dynamics into account.

研究表明，Transformer-XL 语言模型在预测超出其训练期的未来话语时，性能会随着时间的推移而逐渐降低。基于此，提出了动态语言建模的思路，并通过不断更新知识来缓解性能退化问题。因此，应重新思考我们目前培训和评估语言模型的方法，并发展出适应我们日益变化和非稳定的世界的自适应语言模型。