Transformers use the dense self-attention mechanism which gives a lot of
flexibility for long-range connectivity. Over multiple layers of a deep
transformer, the number of possible connectivity patterns increases
exponentially. However, very few of these contribute to the performance of the
network, and even fewer are essential. We hypothesize that there are sparsely
connected sub-networks within a transformer, called information pathways which
can be trained independently. However, the dynamic (i.e., input-dependent)
nature of these pathways makes it difficult to prune dense self-attention
during training. But the overall distribution of these pathways is often
predictable. We take advantage of this fact to propose Stochastically
Subsampled self-Attention (SSA) - a general-purpose training strategy for
transformers that can reduce both the memory and computational cost of
self-attention by 4 to 8 times during training while also serving as a
regularization method - improving generalization over dense training. We show
that an ensemble of sub-models can be formed from the subsampled pathways
within a network, which can achieve better performance than its densely
attended counterpart. We perform experiments on a variety of NLP, computer
vision and graph learning tasks in both generative and discriminative settings
to provide empirical evidence for our claims and show the effectiveness of the
proposed method.

提出一种名为 SSA 的训练策略，该策略可以减少自注意力的内存和计算成本，同时提高泛化能力。该策略基于信息通路的假设，该信息通路可以独立训练，并能形成子模型，优于密集型自注意力模型在不同的 NLP，计算机视觉和图形学任务中表现优异。