BERT (Bidirectional Encoder Representations from Transformers) and related
pre-trained Transformers have provided large gains across many language
understanding tasks, achieving a new state-of-the-art (SOTA). BERT is
pre-trained on two auxiliary tasks: Masked Language Model and Next Sentence
Prediction. In this paper we introduce a new pre-training task inspired by
reading comprehension to better align the pre-training from memorization to
understanding. Span Selection Pre-Training (SSPT) poses cloze-like training
instances, but rather than draw the answer from the model's parameters, it is
selected from a relevant passage. We find significant and consistent
improvements over both BERT-BASE and BERT-LARGE on multiple reading
comprehension (MRC) datasets. Specifically, our proposed model has strong
empirical evidence as it obtains SOTA results on Natural Questions, a new
benchmark MRC dataset, outperforming BERT-LARGE by 3 F1 points on short answer
prediction. We also show significant impact in HotpotQA, improving answer
prediction F1 by 4 points and supporting fact prediction F1 by 1 point and
outperforming the previous best system. Moreover, we show that our pre-training
approach is particularly effective when training data is limited, improving the
learning curve by a large amount.

本篇论文提出了一种新的预训练模型 SSPT，即 Span Selection Pre-Training，通过将预训练模型任务转换为阅读理解，从而提高 BERT 及其他预训练模型的学习能力，实验证明该方法对于数据集有限的场景下表现尤为出色。