The encoder-decoder is the typical framework for Neural Machine Translation
(NMT), and different structures have been developed for improving the
translation performance. Transformer is one of the most promising structures,
which can leverage the self-attention mechanism to capture the semantic
dependency from global view. However, it cannot distinguish the relative
position of different tokens very well, such as the tokens located at the left
or right of the current token, and cannot focus on the local information around
the current token either. To alleviate these problems, we propose a novel
attention mechanism named Hybrid Self-Attention Network (HySAN) which
accommodates some specific-designed masks for self-attention network to extract
various semantic, such as the global/local information, the left/right part
context. Finally, a squeeze gate is introduced to combine different kinds of
SANs for fusion. Experimental results on three machine translation tasks show
that our proposed framework outperforms the Transformer baseline significantly
and achieves superior results over state-of-the-art NMT systems.

本研究中，我们提出了一种名为 HySAN 的新型自注意力机制，它可以通过针对不同类型的自注意力网络设计特定的掩码来提取各种语义，并引入挤压门来融合不同类型的自注意力网络，在三个机器翻译任务上实现了优于 Transform 的显著基线，并实现了超越最先进的 NMT 系统的卓越结果。