Deep neural models (e.g. Transformer) naturally learn spurious features,
which create a ``shortcut'' between the labels and inputs, thus impairing the
generalization and robustness. This paper advances the self-attention mechanism
to its robust variant for Transformer-based pre-trained language models (e.g.
BERT). We propose \textit{Adversarial Self-Attention} mechanism (ASA), which
adversarially biases the attentions to effectively suppress the model reliance
on features (e.g. specific keywords) and encourage its exploration of broader
semantics. We conduct a comprehensive evaluation across a wide range of tasks
for both pre-training and fine-tuning stages. For pre-training, ASA unfolds
remarkable performance gains compared to naive training for longer steps. For
fine-tuning, ASA-empowered models outweigh naive models by a large margin
considering both generalization and robustness.

本研究通过提出对 Transformer 模型的自注意力机制进行改进，提出了 Adversarial Self-Attention（ASA）机制，旨在抑制模型对部分特征的依赖以及探索更广泛的语义，实现更好的泛化性能和鲁棒性。实验结果表明，在预训练和微调阶段中，使用 ASA 的模型相较于普通训练在长远步骤上获得了显著的性能提升，这些模型也能够在泛化性能和鲁棒性方面胜过普通模型。