In this paper, we propose a simple and effective technique to allow for
efficient self-supervised learning with bi-directional Transformers. Our
approach is motivated by recent studies demonstrating that self-attention
patterns in trained models contain a majority of non-linguistic regularities.
We propose a computationally efficient auxiliary loss function to guide
attention heads to conform to such patterns. Our method is agnostic to the
actual pre-training objective and results in faster convergence of models as
well as better performance on downstream tasks compared to the baselines,
achieving state of the art results in low-resource settings. Surprisingly, we
also find that linguistic properties of attention heads are not necessarily
correlated with language modeling performance.

该文提出了一种利用双向 Transformer 实现高效自监督学习的简单而有效的技术，该方法利用辅助损失函数引导注意力头符合自注意力特征，并可以适用于不同的预训练目标，实验证明该方法相对于基线模型更快收敛同时在下游任务中性能更好，在低资源环境中取得了业界领先结果。