Language models (LMs) are typically trained once on a large-scale corpus and
used for years without being updated. However, in a dynamic world, new entities
constantly arise. We propose a framework to analyze what LMs can infer about
new entities that did not exist when the LMs were pretrained. We derive a
dataset of entities indexed by their origination date and paired with their
English Wikipedia articles, from which we can find sentences about each entity.
We evaluate LMs' perplexity on masked spans within these sentences. We show
that models more informed about the entities, such as those with access to a
textual definition of them, achieve lower perplexity on this benchmark. Our
experimental results demonstrate that making inferences about new entities
remains difficult for LMs. Given its wide coverage on entity knowledge and
temporal indexing, our dataset can be used to evaluate LMs and techniques
designed to modify or extend their knowledge. Our automatic data collection
pipeline can be easily used to continually update our benchmark.

本研究提出了一个分析 LMs 推理新实体的框架，并通过使用涵盖实体知识和时间索引的数据集，在句子中评估了 LMs 的困惑度。结果表明，对实体有更多信息的模型在此基准测试中的困惑度更低，但对新实体做出推理仍然很困难。