In the evolving field of Natural Language Processing, understanding the
temporal context of text is increasingly crucial. This study investigates
methods to incorporate temporal information during pre-training, aiming to
achieve effective time-aware language representation for improved performance
on time-related tasks. In contrast to common pre-trained models like BERT,
which rely on synchronic document collections such as BookCorpus and Wikipedia,
our research introduces BiTimeBERT 2.0, a novel language model pre-trained on a
temporal news article collection. BiTimeBERT 2.0 utilizes this temporal news
collection, focusing on three innovative pre-training objectives: Time-Aware
Masked Language Modeling (TAMLM), Document Dating (DD), and Time-Sensitive
Entity Replacement (TSER). Each objective targets a unique aspect of temporal
information. TAMLM is designed to enhance the understanding of temporal
contexts and relations, DD integrates document timestamps as chronological
markers, and TSER focuses on the temporal dynamics of "Person" entities,
recognizing their inherent temporal significance. The experimental results
consistently demonstrate that BiTimeBERT 2.0 outperforms models like BERT and
other existing pre-trained models, achieving substantial gains across a variety
of downstream NLP tasks and applications where time plays a pivotal role.

本研究探讨了在自然语言处理领域中理解文本的时间背景的方法，并介绍了一种新型语言模型 BiTimeBERT 2.0，该模型通过预训练在一个时间新闻文章集合上，利用三种创新的预训练目标以有效地获得时间感知的语言表示，从而在与时间相关的任务中实现改进的性能。实验结果显示，BiTimeBERT 2.0 在多种依赖时间的下游自然语言处理任务和应用中显著优于 BERT 和其他现有的预训练模型。