While the self-attention mechanism has been widely used in a wide variety of
tasks, it has the unfortunate property of a quadratic cost with respect to the
input length, which makes it difficult to deal with long inputs. In this paper,
we present a method for accelerating and structuring self-attentions: Sparse
Adaptive Connection (SAC). In SAC, we regard the input sequence as a graph and
attention operations are performed between linked nodes. In contrast with
previous self-attention models with pre-defined structures (edges), the model
learns to construct attention edges to improve task-specific performances. In
this way, the model is able to select the most salient nodes and reduce the
quadratic complexity regardless of the sequence length. Based on SAC, we show
that previous variants of self-attention models are its special cases. Through
extensive experiments on neural machine translation, language modeling, graph
representation learning and image classification, we demonstrate SAC is
competitive with state-of-the-art models while significantly reducing memory
cost.

本文介绍 Sparse Adaptive Connection（SAC）方法，将输入序列视为图，并通过构建关注边，以改进任务特定性能，优化了自注意力机制。通过在图形表示学习和图像分类中实施的广泛实验，证明了 SAC 在减少内存成本的同时，可以与最先进的模型竞争。