Masking tokens uniformly at random constitutes a common flaw in the
pretraining of Masked Language Models (MLMs) such as BERT. We show that such
uniform masking allows an MLM to minimize its training objective by latching
onto shallow local signals, leading to pretraining inefficiency and suboptimal
downstream performance. To address this flaw, we propose PMI-Masking, a
principled masking strategy based on the concept of Pointwise Mutual
Information (PMI), which jointly masks a token n-gram if it exhibits high
collocation over the corpus. PMI-Masking motivates, unifies, and improves upon
prior more heuristic approaches that attempt to address the drawback of random
uniform token masking, such as whole-word masking, entity/phrase masking, and
random-span masking. Specifically, we show experimentally that PMI-Masking
reaches the performance of prior masking approaches in half the training time,
and consistently improves performance at the end of training.

通过对点对互信息的概念提出了 PMI-Masking，该结构代替了以往不合理的随机掩码方法，实验结果表明在预训练的效率和下游任务性能上有所提高

基于 PMI 的相关跨度掩码

PMI-Masking: Principled masking of correlated spans

For natural language processing `text-to-text' tasks, the prevailing
approaches heavily rely on pretraining large self-supervised models on
increasingly larger `task-external' data. Transfer learning from high-resource
pretraining works well, but research has focused on settings with very large
data and compute requirements, while the potential of efficient low-resource
learning, without large `task-external' pretraining, remains under-explored. In
this work, we evaluate against three core challenges for resource efficient
learning. Namely, we analyze: (1) pretraining data ($X$) efficiency; (2) zero
to few-shot label ($Y$) efficiency; and (3) long-tail generalization, since
long-tail preservation has been linked to algorithmic fairness and because data
in the tail is limited by definition. To address these challenges, we propose a
data and compute efficient self-supervised, contrastive text encoder,
pretrained on 60MB of `task-internal' text data, and compare it to RoBERTa,
which was pretrained on 160GB of `task-external' text. We find our method
outperforms RoBERTa, while pretraining and fine-tuning in a 1/5th of RoBERTa's
fine-tuning time.

本文提出了一种基于对比学习的文本编码器，利用 60MB 的任务内文本数据进行预训练，相较于利用 160GB 的任务外文本进行预训练的 RoBERTa 模型，在预训练效率、数据和计算资源利用率、以及算法公正性方面有了显著提高。