Self-attention networks (SANs) with selective mechanism has produced
substantial improvements in various NLP tasks by concentrating on a subset of
input words. However, the underlying reasons for their strong performance have
not been well explained. In this paper, we bridge the gap by assessing the
strengths of selective SANs (SSANs), which are implemented with a flexible and
universal Gumbel-Softmax. Experimental results on several representative NLP
tasks, including natural language inference, semantic role labelling, and
machine translation, show that SSANs consistently outperform the standard SANs.
Through well-designed probing experiments, we empirically validate that the
improvement of SSANs can be attributed in part to mitigating two commonly-cited
weaknesses of SANs: word order encoding and structure modeling. Specifically,
the selective mechanism improves SANs by paying more attention to content words
that contribute to the meaning of the sentence. The code and data are released
at this https URL

本文通过评估实现灵活通用的 Gumbel-Softmax 的选择性自注意力网络（SSANs）的性能，在多个代表性 NLP 任务中，包括自然语言推理、语义角色标注和机器翻译，实验结果显示，SSANs 一致优于标准 SANs 并通过良好设计的探针实验证明，其改进性在一定程度上归因于缓解 SANs 的两个常见缺点：词序编码和结构建模，具体而言，选择机制通过更多地关注对句子含义有贡献的内容单词，提高了 SANs 的性能。