In this paper, we propose a novel architecture, the Enhanced Interactive
Transformer (EIT), to address the issue of head degradation in self-attention
mechanisms. Our approach replaces the traditional multi-head self-attention
mechanism with the Enhanced Multi-Head Attention (EMHA) mechanism, which
relaxes the one-to-one mapping constraint among queries and keys, allowing each
query to attend to multiple keys. Furthermore, we introduce two interaction
models, Inner-Subspace Interaction and Cross-Subspace Interaction, to fully
utilize the many-to-many mapping capabilities of EMHA. Extensive experiments on
a wide range of tasks (e.g. machine translation, abstractive summarization,
grammar correction, language modelling and brain disease automatic diagnosis)
show its superiority with a very modest increase in model size.

本文提出了一种新的神经网络架构，即增强交互式 Transformer (EIT)，用于解决自注意机制中的头部衰减问题。我们将传统的多头自注意机制替换为增强的多头注意力机制 (EMHA)，并引入两种交互模型，Inner-Subspace 交互和 Cross-Subspace 交互，以充分利用 EMHA 的映射能力。通过广泛的实验，我们发现 EIT 在多项任务上 (如机器翻译、摘要生成、语法纠正、语言建模和脑疾病自动诊断)，仅仅在模型大小方面有很小的增加就能以优异的性能超越传统模型。