A simple design recipe for deep Transformers is to compose identical building
blocks. But standard transformer blocks are far from simple, interweaving
attention and MLP sub-blocks with skip connections & normalisation layers in
precise arrangements. This complexity leads to brittle architectures, where
seemingly minor changes can significantly reduce training speed, or render
models untrainable.
In this work, we ask to what extent the standard transformer block can be
simplified? Combining signal propagation theory and empirical observations, we
motivate modifications that allow many block components to be removed with no
loss of training speed, including skip connections, projection or value
parameters, sequential sub-blocks and normalisation layers. In experiments on
both autoregressive decoder-only and BERT encoder-only models, our simplified
transformers emulate the per-update training speed and performance of standard
transformers, while enjoying 15% faster training throughput, and using 15%
fewer parameters.

通过信号传播理论和经验观察，我们提出了修改的方案，可以简化标准的 Transformer 模块，包括去除跳跃连接、投影或值参数、顺序子模块和归一化层等组件，而无需牺牲训练速度，实验证明这种简化版本的 Transformer 实现了标准 Transformer 的更新速度和性能，同时训练吞吐量提高了 15％，参数数量减少了 15％。