While pretrained language models ("LM") have driven impressive gains over
morpho-syntactic and semantic tasks, their ability to model discourse and
pragmatic phenomena is less clear. As a step towards a better understanding of
their discourse modelling capabilities, we propose a sentence intrusion
detection task. We examine the performance of a broad range of pretrained LMs
on this detection task for English. Lacking a dataset for the task, we
introduce INSteD, a novel intruder sentence detection dataset, containing
170,000+ documents constructed from English Wikipedia and CNN news articles.
Our experiments show that pretrained LMs perform impressively in in-domain
evaluation, but experience a substantial drop in the cross-domain setting,
indicating limited generalisation capacity. Further results over a novel
linguistic probe dataset show that there is substantial room for improvement,
especially in the cross-domain setting.

在理解预训练语言模型对话建模能力方面，我们提出了一种句子入侵检测任务，并在英语方面检查了一系列预训练 LM 的性能。我们通过构建包含英语维基百科和 CNN 新闻文章的 170,000 + 文档的新型入侵句子检测数据集 INSteD，显示预训练 LM 在域内评估中表现出色，但在跨域设置中经验了大幅下降，表明其对跨域推广的能力有限。进一步的，在一个新的语言探测数据集上的结果表明，在跨域设置中有很大的改进空间。