We introduce Sentence-level Language Modeling, a new pre-training objective
for learning a discourse language representation in a fully self-supervised
manner. Recent pre-training methods in NLP focus on learning either bottom or
top-level language representations: contextualized word representations derived
from language model objectives at one extreme and a whole sequence
representation learned by order classification of two given textual segments at
the other. However, these models are not directly encouraged to capture
representations of intermediate-size structures that exist in natural languages
such as sentences and the relationships among them. To that end, we propose a
new approach to encourage learning of a contextualized sentence-level
representation by shuffling the sequence of input sentences and training a
hierarchical transformer model to reconstruct the original ordering. Through
experiments on downstream tasks such as GLUE, SQuAD, and DiscoEval, we show
that this feature of our model improves the performance of the original BERT by
large margins.

通过自我监督的方式，我们提出了句子级别的语言建模作为新的预训练目标，通过打乱输入句子的顺序训练用于重构原始顺序的分层 transformer 模型以鼓励中型结构的学习，进而提高原 BERT 模型的性能。

SLM：使用句子重排学习语篇语言表示

SLM: Learning a Discourse Language Representation with Sentence  Unshuffling

This paper examines the challenging problem of learning representations of
entities and relations in a complex multi-relational knowledge graph. We
propose HittER, a Hierarchical Transformer model to jointly learn
Entity-relation composition and Relational contextualization based on a source
entity's neighborhood. Our proposed model consists of two different Transformer
blocks: the bottom block extracts features of each entity-relation pair in the
local neighborhood of the source entity and the top block aggregates the
relational information from outputs of the bottom block. We further design a
masked entity prediction task to balance information from the relational
context and the source entity itself. Experimental results show that HittER
achieves new state-of-the-art results on multiple link prediction datasets. We
additionally propose a simple approach to integrate HittER into BERT and
demonstrate its effectiveness on two Freebase factoid question answering
datasets.

本文针对多关系知识图谱中实体和关系的学习表示问题，提出了一种基于分层 Transformer 模型的方法，该方法可以联合学习实体关系组合和基于源实体社区的关系上下文。实验结果表明 HittER 模型在多个链接预测数据集上取得了最新的最优结果。此外，我们还提出了一种简单的方法将 HittER 集成到 BERT，并在两个 Freebase 问答数据集上证明了其有效性。