Recurrent neural networks (RNN), convolutional neural networks (CNN) and
self-attention networks (SAN) are commonly used to produce context-aware
representations. RNN can capture long-range dependency but is hard to
parallelize and not time-efficient. CNN focuses on local dependency but does
not perform well on some tasks. SAN can model both such dependencies via highly
parallelizable computation, but memory requirement grows rapidly in line with
sequence length. In this paper, we propose a model, called "bi-directional
block self-attention network (Bi-BloSAN)", for RNN/CNN-free sequence encoding.
It requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN
splits the entire sequence into blocks, and applies an intra-block SAN to each
block for modeling local context, then applies an inter-block SAN to the
outputs for all blocks to capture long-range dependency. Thus, each SAN only
needs to process a short sequence, and only a small amount of memory is
required. Additionally, we use feature-level attention to handle the variation
of contexts around the same word, and use forward/backward masks to encode
temporal order information. On nine benchmark datasets for different NLP tasks,
Bi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better
efficiency-memory trade-off than existing RNN/CNN/SAN.

本文提出了一种双向块自注意力网络 (Bi-BloSAN) 来进行 RNN/CNN-free 序列编码，用于解决 RNN/CNN/SAN 的时间效率和内存开销问题，并在 NLP 任务中取得了最新水平的表现。