This work presents an analysis of the effectiveness of using standard shallow
feed-forward networks to mimic the behavior of the attention mechanism in the
original Transformer model, a state-of-the-art architecture for
sequence-to-sequence tasks. We substitute key elements of the attention
mechanism in the Transformer with simple feed-forward networks, trained using
the original components via knowledge distillation. Our experiments, conducted
on the IWSLT2017 dataset, reveal the capacity of these "attentionless
Transformers" to rival the performance of the original architecture. Through
rigorous ablation studies, and experimenting with various replacement network
types and sizes, we offer insights that support the viability of our approach.
This not only sheds light on the adaptability of shallow feed-forward networks
in emulating attention mechanisms but also underscores their potential to
streamline complex architectures for sequence-to-sequence tasks.

本研究通过分析使用标准的浅层前馈网络来模拟原始 Transformer 模型中的注意力机制的有效性，采用知识蒸馏的方法，用简单的前馈网络替换 Transformer 中的关键元素，并在 IWSLT2017 数据集上进行实验，发现这种 “无注意力 Transformer” 具备与原始架构相媲美的性能，通过深入的消融研究和尝试不同的替代网络类型和规模，我们提供了支持我们方法可行性的见解，这不仅揭示了浅层前馈网络在模拟注意力机制方面的适应性，也强调了它们简化序列任务复杂架构的潜力。

重新思考注意力：探索浅层前馈神经网络作为 Transformer 中注意力层的替代方案

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as  an Alternative to Attention Layers in Transformers

Currently, deep neural networks are the state of the art on problems such as
speech recognition and computer vision. In this extended abstract, we show that
shallow feed-forward networks can learn the complex functions previously
learned by deep nets and achieve accuracies previously only achievable with
deep models. Moreover, in some cases the shallow neural nets can learn these
deep functions using a total number of parameters similar to the original deep
model. We evaluate our method on the TIMIT phoneme recognition task and are
able to train shallow fully-connected nets that perform similarly to complex,
well-engineered, deep convolutional architectures. Our success in training
shallow neural nets to mimic deeper models suggests that there probably exist
better algorithms for training shallow feed-forward nets than those currently
available.

该研究表明，浅层前馈神经网络可以学习先前由深层网络学习的复杂函数，并且可以达到仅用深度模型可以实现的准确度，在 TIMIT 语音识别任务中，没有经过复杂设计的浅层神经网络能够表现类似于深度卷积网络，并且成功地训练浅层神经网络模仿更深层次模型的方法表明，可能存在比当前更好的训练浅层前馈网络的算法。