We introduce Sentence-level Language Modeling, a new pre-training objective
for learning a discourse language representation in a fully self-supervised
manner. Recent pre-training methods in NLP focus on learning either bottom or
top-level language representations: contextualized word representations derived
from language model objectives at one extreme and a whole sequence
representation learned by order classification of two given textual segments at
the other. However, these models are not directly encouraged to capture
representations of intermediate-size structures that exist in natural languages
such as sentences and the relationships among them. To that end, we propose a
new approach to encourage learning of a contextualized sentence-level
representation by shuffling the sequence of input sentences and training a
hierarchical transformer model to reconstruct the original ordering. Through
experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show
that this feature of our model improves the performance of the original BERT by
large margins.

通过自我监督的方式，我们提出了句子级别的语言建模作为新的预训练目标，通过打乱输入句子的顺序训练用于重构原始顺序的分层 transformer 模型以鼓励中型结构的学习，进而提高原 BERT 模型的性能。