Dynamic contextualised word embeddings represent the temporal semantic
variations of words. We propose a method for learning dynamic contextualised
word embeddings by time-adapting a pretrained Masked Language Model (MLM) using
time-sensitive templates. Given two snapshots $C_1$ and $C_2$ of a corpora
taken respectively at two distinct timestamps $T_1$ and $T_2$, we first propose
an unsupervised method to select (a) pivot terms related to both $C_1$ and
$C_2$, and (b) anchor terms that are associated with a specific pivot term in
each individual snapshot. We then generate prompts by filling manually compiled
templates using the extracted pivot and anchor terms. Moreover, we propose an
automatic method to learn time-sensitive templates from $C_1$ and $C_2$,
without requiring any human supervision. Next, we use the generated prompts to
adapt a pretrained MLM to $T_2$ by fine-tuning it on the prompts. Experimental
results show that our proposed method significantly reduces the perplexity of
test sentences selected from $T_2$, thereby outperforming the current
state-of-the-art dynamic contextualised word embedding methods.

本文提出了一种通过时间感知模板来学习动态上下文词嵌入的方法，实验结果表明该方法比当前最先进的动态上下文词嵌入方法表现更好。