We propose an efficient interactive method for multi-head self-attention via
decomposition. For existing methods using multi-head self-attention, the
attention operation of each head is computed independently. However, we show
that the interactions between cross-heads of the attention matrix enhance the
information flow of the attention operation. Considering that the attention
matrix of each head can be seen as a feature of networks, it is beneficial to
establish connectivity between them to capture interactions better. However, a
straightforward approach to capture the interactions between the cross-heads is
computationally prohibitive as the complexity grows substantially with the high
dimension of an attention matrix. In this work, we propose an effective method
to decompose the attention operation into query- and key-less components. This
will result in a more manageable size for the attention matrix, specifically
for the cross-head interactions. Expensive experimental results show that the
proposed cross-head interaction approach performs favorably against existing
efficient attention methods and state-of-the-art backbone models.

我们通过分解提出了一种高效的多头自注意力交互方法，该方法通过建立注意力矩阵中的跨头交互来增强信息流动，并将注意力操作分解成查询和无键组件，以降低注意力矩阵的维度，从而实现更高效的注意力操作。实验结果表明，所提出的跨头交互方法在效率和性能上优于现有的方法和最先进的骨干模型。

具有线性复杂度的交互式多头自注意力

Interactive Multi-Head Self-Attention with Linear Complexity

Document-level machine translation leverages inter-sentence dependencies to
produce more coherent and consistent translations. However, these models,
predominantly based on transformers, are difficult to scale to long documents
as their attention layers have quadratic complexity in the sequence length.
Recent efforts on efficient attention improve scalability, but their effect on
document translation remains unexplored. In this work, we investigate the
efficacy of a recent linear attention model by Peng et al. (2021) on document
translation and augment it with a sentential gate to promote a recency
inductive bias. We evaluate the model on IWSLT 2015 and OpenSubtitles 2018
against the transformer, demonstrating substantially increased decoding speed
on long sequences with similar or better BLEU scores. We show that sentential
gating further improves translation quality on IWSLT.

本文研究了一种基于 Peng et al. (2021) 的线性注意力模型，并通过句子门促进最近归纳偏置，以提高文档级机器翻译的效能和翻译质量，并在 IWSLT 2015 和 OpenSubtitles 2018 上进行了评估和测试，结果表明该模型在长序列的解码速度方面有大幅提高，且翻译得分相似或更高，并展示了句子门进一步提高了 IWSLT 上的翻译质量。