Self-attention networks have proven to be of profound value for its strength
of capturing global dependencies. In this work, we propose to model localness
for self-attention networks, which enhances the ability of capturing useful
local context. We cast localness modeling as a learnable Gaussian bias, which
indicates the central and scope of the local region to be paid more attention.
The bias is then incorporated into the original attention distribution to form
a revised distribution. To maintain the strength of capturing long distance
dependencies and enhance the ability of capturing short-range dependencies, we
only apply localness modeling to lower layers of self-attention networks.
Quantitative and qualitative analyses on Chinese-English and English-German
translation tasks demonstrate the effectiveness and universality of the
proposed approach.

本篇研究提出了一种基于学习性高斯偏置的自注意力网络的局部特征建模方法，能够从更细致的角度获取有用的局部上下文信息，通过实验验证证明该方法能够有效地应用于翻译任务中。

自注意力网络的本地性建模

Modeling Localness for Self-Attention Networks

Recurrent neural networks (RNN), convolutional neural networks (CNN) and
self-attention networks (SAN) are commonly used to produce context-aware
representations. RNN can capture long-range dependency but is hard to
parallelize and not time-efficient. CNN focuses on local dependency but does
not perform well on some tasks. SAN can model both such dependencies via highly
parallelizable computation, but memory requirement grows rapidly in line with
sequence length. In this paper, we propose a model, called "bi-directional
block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding.
It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN
splits the entire sequence into blocks, and applies an intra-block SAN to each
block for modeling local context, then applies an inter-block SAN to the
outputs for all blocks to capture long-range dependency. Thus, each SAN only
needs to process a short sequence, and only a small amount of memory is
required. Additionally, we use feature-level attention to handle the variation
of contexts around the same word, and use forward/backward masks to encode
temporal order information. On nine benchmark datasets for different NLP tasks,
Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better
efficiency-memory trade-off than existing RNN/CNN/SAN.

本文提出了一种双向块自注意力网络 (Bi-BloSAN) 来进行 RNN/CNN-free 序列编码，用于解决 RNN/CNN/SAN 的时间效率和内存开销问题，并在 NLP 任务中取得了最新水平的表现。