Temporal concept drift refers to the problem of data changing over time. In
NLP, that would entail that language (e.g. new expressions, meaning shifts) and
factual knowledge (e.g. new concepts, updated facts) evolve over time. Focusing
on the latter, we benchmark $11$ pretrained masked language models (MLMs) on a
series of tests designed to evaluate the effect of temporal concept drift, as
it is crucial that widely used language models remain up-to-date with the
ever-evolving factual updates of the real world. Specifically, we provide a
holistic framework that (1) dynamically creates temporal test sets of any time
granularity (e.g. month, quarter, year) of factual data from Wikidata, (2)
constructs fine-grained splits of tests (e.g. updated, new, unchanged facts) to
ensure comprehensive analysis, and (3) evaluates MLMs in three distinct ways
(single-token probing, multi-token generation, MLM scoring). In contrast to
prior work, our framework aims to unveil how robust an MLM is over time and
thus to provide a signal in case it has become outdated, by leveraging multiple
views of evaluation.

该研究在 Wikidata 数据集上，评估了 11 个预训练的蒙面语言模型对时间概念漂移的影响，以确保其能够适应不断更新的现实世界的事实更新，并提供了一个综合框架来构建不同时间粒度的事实数据集，并通过多个评估角度来评估模型是否过时。