Current language models are usually trained using a self-supervised scheme,
where the main focus is learning representations at the word or sentence level.
However, there has been limited progress in generating useful discourse-level
representations. In this work, we propose to use ideas from predictive coding
theory to augment BERT-style language models with a mechanism that allows them
to learn suitable discourse-level representations. As a result, our proposed
approach is able to predict future sentences using explicit top-down
connections that operate at the intermediate layers of the network. By
experimenting with benchmarks designed to evaluate discourse-related knowledge
using pre-trained sentence representations, we demonstrate that our approach
improves performance in 6 out of 11 tasks by excelling in discourse
relationship detection.

本文将预测编码理论引入 BERT 风格语言模型，以学习适当的语篇水平表示，实现了通过中间层进行顶部到底部的明显连接预测未来的句子，从而在语篇相关知识的基准测试中提高性能，特别是在语篇关系检测方面。

利用预测编码增强 BERT 模型以提高语篇级别表征

Augmenting BERT-style Models with Predictive Coding to Improve  Discourse-level Representations

Recent models for unsupervised representation learning of text have employed
a number of techniques to improve contextual word representations but have put
little focus on discourse-level representations. We propose CONPONO, an
inter-sentence objective for pretraining language models that models discourse
coherence and the distance between sentences. Given an anchor sentence, our
model is trained to predict the text k sentences away using a sampled-softmax
objective where the candidates consist of neighboring sentences and sentences
randomly sampled from the corpus. On the discourse representation benchmark
DiscoEval, our model improves over the previous state-of-the-art by up to 13%
and on average 4% absolute across 7 tasks. Our model is the same size as
BERT-Base, but outperforms the much larger BERT- Large model and other more
recent approaches that incorporate discourse. We also show that CONPONO yields
gains of 2%-6% absolute even for tasks that do not explicitly evaluate
discourse: textual entailment (RTE), common sense reasoning (COPA) and reading
comprehension (ReCoRD).

CONPONO 提供了一种跨句子的目标预训练语言模型，该模型在表示学习中模拟语篇连贯性和句子之间的距离， 并在 DiscoEval 基准测试中取得了 13％的改进。