This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of the original architecture. Through rigorous ablation studies, and experimenting with various replacement network types and sizes, we offer insights that support the viability of our approach. This not only sheds light on the adaptability of shallow feed-forward networks in emulating attention mechanisms but also underscores their potential to streamline complex architectures for sequence-to-sequence tasks.

本研究通过分析使用标准的浅层前馈网络来模拟原始Transformer模型中的注意力机制的有效性，采用知识蒸馏的方法，用简单的前馈网络替换Transformer中的关键元素，并在IWSLT2017数据集上进行实验，发现这种“无注意力Transformer”具备与原始架构相媲美的性能，通过深入的消融研究和尝试不同的替代网络类型和规模，我们提供了支持我们方法可行性的见解，这不仅揭示了浅层前馈网络在模拟注意力机制方面的适应性，也强调了它们简化序列任务复杂架构的潜力。

重新思考注意力：探索浅层前馈神经网络作为Transformer中注意力层的替代方案