This study aims to acquire more insights into the continuous pre-training
phase of BERT regarding entity knowledge, using the COVID-19 pandemic as a case
study. Since the pandemic emerged after the last update of BERT's pre-training
data, the model has little to no entity knowledge about COVID-19. Using
continuous pre-training, we control what entity knowledge is available to the
model. We compare the baseline BERT model with the further pre-trained variants
on the fact-checking benchmark Check-COVID. To test the robustness of
continuous pre-training, we experiment with several adversarial methods to
manipulate the input data, such as training on misinformation and shuffling the
word order until the input becomes nonsensical. Surprisingly, our findings
reveal that these methods do not degrade, and sometimes even improve, the
model's downstream performance. This suggests that continuous pre-training of
BERT is robust against misinformation. Furthermore, we are releasing a new
dataset, consisting of original texts from academic publications in the
LitCovid repository and their AI-generated false counterparts.

研究利用 COVID-19 作为案例，探讨 BERT 的持续预训练阶段，尤其是实体知识的获取。结果表明，持续预训练使得 BERT 对 COVID-19 具有一定的实体知识。在使用事实核查基准 Check-COVID 测试连续预训练的鲁棒性时，我们发现恶意训练和词序乱序等故意篡改数据的方法并没有降低模型的下游性能，甚至有时还能改善，这表明 BERT 的持续预训练对于恶意信息具有一定的鲁棒性。此外，我们还发布了一个新数据集，包括 LitCovid 仓库中的原始文本和 AI 生成的错误对应文本。

谎言的集袋：BERT 在持续预训练中的鲁棒性

Bag of Lies: Robustness in Continuous Pre-training BERT

Previous works show that Pre-trained Language Models (PLMs) can capture
factual knowledge. However, some analyses reveal that PLMs fail to perform it
robustly, e.g., being sensitive to the changes of prompts when extracting
factual knowledge. To mitigate this issue, we propose to let PLMs learn the
deterministic relationship between the remaining context and the masked
content. The deterministic relationship ensures that the masked factual content
can be deterministically inferable based on the existing clues in the context.
That would provide more stable patterns for PLMs to capture factual knowledge
than randomly masking. Two pre-training tasks are further introduced to
motivate PLMs to rely on the deterministic relationship when filling masks.
Specifically, we use an external Knowledge Base (KB) to identify deterministic
relationships and continuously pre-train PLMs with the proposed methods. The
factual knowledge probing experiments indicate that the continuously
pre-trained PLMs achieve better robustness in factual knowledge capturing.
Further experiments on question-answering datasets show that trying to learn a
deterministic relationship with the proposed methods can also help other
knowledge-intensive tasks.

该论文提出了一种基于确定性关系的预训练方法，利用外部知识库识别确定性关系并连续预训练语言模型以提高其在提取事实知识方面的鲁棒性，试图学习与之相关的确定性关系有助于知识密集型任务。

使用确定性事实知识进行语言模型预训练

Pre-training Language Models with Deterministic Factual Knowledge

Recently, prompt tuning \cite{lester2021power} has gradually become a new
paradigm for NLP, which only depends on the representation of the words by
freezing the parameters of pre-trained language models (PLMs) to obtain
remarkable performance on downstream tasks. It maintains the consistency of
Masked Language Model (MLM) \cite{devlin2018bert} task in the process of
pre-training, and avoids some issues that may happened during fine-tuning.
Naturally, we consider that the "[MASK]" tokens carry more useful information
than other tokens because the model combines with context to predict the masked
tokens. Among the current prompt tuning methods, there will be a serious
problem of random composition of the answer tokens in prediction when they
predict multiple words so that they have to map tokens to labels with the help
verbalizer. In response to the above issue, we propose a new \textbf{Pro}mpt
\textbf{Tu}ning based on "[\textbf{M}ASK]" (\textbf{Protum}) method in this
paper, which constructs a classification task through the information carried
by the hidden layer of "[MASK]" tokens and then predicts the labels directly
rather than the answer tokens. At the same time, we explore how different
hidden layers under "[MASK]" impact on our classification model on many
different data sets. Finally, we find that our \textbf{Protum} can achieve much
better performance than fine-tuning after continuous pre-training with less
time consumption. Our model facilitates the practical application of large
models in NLP.

本文提出了一种基于 'MASK' token 编码的 Prompt Tuning 方法 ——Protum，通过隐藏层信息构造分类任务并直接预测标签，相较于 fine-tuning，该方法在较短的时间内连续预训练后取得了更好的性能。