Contextualized word embeddings have demonstrated state-of-the-art performance
in various natural language processing tasks including those that concern
historical semantic change. However, language models such as BERT was trained
primarily on contemporary corpus data. To investigate whether training on
historical corpus data improves diachronic semantic analysis, we present a
pre-trained BERT-based language model, HistBERT, trained on the balanced Corpus
of Historical American English. We examine the effectiveness of our approach by
comparing the performance of the original BERT and that of HistBERT, and we
report promising results in word similarity and semantic shift analysis. Our
work suggests that the effectiveness of contextual embeddings in diachronic
semantic analysis is dependent on the temporal profile of the input text and
care should be taken in applying this methodology to study historical semantic
change.

研究历史语料库数据对语言模型 BERT 训练的影响，从而提出一种预先训练的基于 HistBERT 的语言模型，并比较它与原始 BERT 在单词相似度和语义转化分析方面的表现，该工作强调，上下文语言嵌入在历史语言分析中的效果取决于输入文本的时态特征，并应谨慎应用此方法来研究历史语义变化。