Transformer is the backbone of modern NLP models. In this paper, we propose
RealFormer, a simple and generic technique to create Residual Attention Layer
Transformer networks that significantly outperform the canonical Transformer
and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked
Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA,
Natural Questions, and OpenKP. We also observe empirically that RealFormer
stabilizes training and leads to models with sparser attention. Source code and
pre-trained checkpoints for RealFormer can be found at
this https URL

提出了一种名为 RealFormer 的技术，可以创建残差注意力层的 Transformer 网络，并在许多任务上显著优于标准 Transformer 及其变体。该技术不仅稳定了训练，而且还可以导致具有更少注意力的模型，并提供了预训练检查点。

RealFormer: 喜欢残差注意力的 Transformer

RealFormer: Transformer Likes Residual Attention

Unsupervised pre-training of large neural models has recently revolutionized
Natural Language Processing. By warm-starting from the publicly released
checkpoints, NLP practitioners have pushed the state-of-the-art on multiple
benchmarks while saving significant amounts of compute time. So far the focus
has been mainly on the Natural Language Understanding tasks. In this paper, we
demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. We
developed a Transformer-based sequence-to-sequence model that is compatible
with publicly available pre-trained BERT, GPT-2 and RoBERTa checkpoints and
conducted an extensive empirical study on the utility of initializing our
model, both encoder and decoder, with these checkpoints. Our models result in
new state-of-the-art results on Machine Translation, Text Summarization,
Sentence Splitting, and Sentence Fusion.

本篇论文展示了预训练模型的高效性，使用 Transformer-based 序列到序列模型，并将其与公开的预训练检查点匹配，取得了机器翻译、文本摘要、句子拆分和句子融合等方面的最新的最新成果。