Current language models are usually trained using a self-supervised scheme,
where the main focus is learning representations at the word or sentence level.
However, there has been limited progress in generating useful discourse-level
representations. In this work, we propose to use ideas from predictive coding
theory to augment BERT-style language models with a mechanism that allows them
to learn suitable discourse-level representations. As a result, our proposed
approach is able to predict future sentences using explicit top-down
connections that operate at the intermediate layers of the network. By
experimenting with benchmarks designed to evaluate discourse-related knowledge
using pre-trained sentence representations, we demonstrate that our approach
improves performance in 6 out of 11 tasks by excelling in discourse
relationship detection.

本文将预测编码理论引入 BERT 风格语言模型，以学习适当的语篇水平表示，实现了通过中间层进行顶部到底部的明显连接预测未来的句子，从而在语篇相关知识的基准测试中提高性能，特别是在语篇关系检测方面。