Large-scale learning of transformer language models has yielded improvements
on a variety of natural language understanding tasks. Whether they can be
effectively adapted for summarization, however, has been less explored, as the
learned representations are less seamlessly integrated into existing neural
text production architectures. In this work, we propose two solutions for
efficiently adapting pretrained transformer language models as text
summarizers: source embeddings and domain-adaptive training. We test these
solutions on three abstractive summarization datasets, achieving new state of
the art performance on two of them. Finally, we show that these improvements
are achieved by producing more focused summaries with fewer superfluous and
that performance improvements are more pronounced on more abstractive datasets.

本文探讨使用预训练的 Transformer 语言模型来进行文本摘要的实现，提出了基于源嵌入和领域自适应训练的方法，并在三个摘要数据集上进行了测试，并在其中两个数据集上取得了新的最佳表现。结果表明，该方法能够产生更专注的摘要，并且对于更抽象的数据集表现得更加明显。