Attention mechanism has been used as an ancillary means to help RNN or CNN.
However, the Transformer (Vaswani et al., 2017) recently recorded the
state-of-the-art performance in machine translation with a dramatic reduction
in training time by solely using attention. Motivated by the Transformer,
Directional Self Attention Network (Shen et al., 2017), a fully attention-based
sentence encoder, was proposed. It showed good performance with various data by
using forward and backward directional information in a sentence. But in their
study, not considered at all was the distance between words, an important
feature when learning the local dependency to help understand the context of
input text. We propose Distance-based Self-Attention Network, which considers
the word distance by using a simple distance mask in order to model the local
dependency without losing the ability of modeling global dependency which
attention has inherent. Our model shows good performance with NLI data, and it
records the new state-of-the-art result with SNLI data. Additionally, we show
that our model has a strength in long sentences or documents.

本研究提出了一种基于距离的自注意力网络模型，利用简单的距离掩码考虑单词之间的距离，以便在不失去全局信息依赖的情况下建立本地依赖关系，并在 NLI 数据方面表现出良好的性能，同时长文档也有优势；而之前的基于注意力机制的模型，则没有考虑到单词之间的距离。