A simple design recipe for deep Transformers is to compose identical building
blocks. But standard transformer blocks are far from simple, interweaving
attention and MLP sub-blocks with skip connections & normalisation layers in
precise arrangements. This complexity leads to brittle architectures, where
seemingly minor changes can significantly reduce training speed, or render
models untrainable.
In this work, we ask to what extent the standard transformer block can be
simplified? Combining signal propagation theory and empirical observations, we
motivate modifications that allow many block components to be removed with no
loss of training speed, including skip connections, projection or value
parameters, sequential sub-blocks and normalisation layers. In experiments on
both autoregressive decoder-only and BERT encoder-only models, our simplified
transformers emulate the per-update training speed and performance of standard
transformers, while enjoying 15% faster training throughput, and using 15%
fewer parameters.

通过信号传播理论和经验观察，我们提出了修改的方案，可以简化标准的 Transformer 模块，包括去除跳跃连接、投影或值参数、顺序子模块和归一化层等组件，而无需牺牲训练速度，实验证明这种简化版本的 Transformer 实现了标准 Transformer 的更新速度和性能，同时训练吞吐量提高了 15％，参数数量减少了 15％。

简化 Transformer 块

Simplifying Transformer Blocks

Syllabification does not seem to improve word-level RNN language modeling
quality when compared to character-based segmentation. However, our best
syllable-aware language model, achieving performance comparable to the
competitive character-aware model, has 18%-33% fewer parameters and is trained
1.2-2.2 times faster.

比较音节划分和基于字符划分，在词级 RNN 语言建模的质量提高方面效果不明显。 然而，我们最好的音节感知语言模型表现出与竞争性基于字符模型相当的性能，参数少了 18％-33％，并且训练速度提高了 1.2-2.2 倍。

音节感知的神经语言模型：无法打败字符感知的模型

Syllable-aware Neural Language Models: A Failure to Beat Character-aware  Ones

Convolutional Neural Networks (CNNs) have been successfully applied to many
computer vision tasks, such as image classification. By performing linear
combinations and element-wise nonlinear operations, these networks can be
thought of as extracting solely first-order information from an input image. In
the past, however, second-order statistics computed from handcrafted features,
e.g., covariances, have proven highly effective in diverse recognition tasks.
In this paper, we introduce a novel class of CNNs that exploit second-order
statistics. To this end, we design a series of new layers that (i) extract a
covariance matrix from convolutional activations, (ii) compute a parametric,
second-order transformation of a matrix, and (iii) perform a parametric
vectorization of a matrix. These operations can be assembled to form a
Covariance Descriptor Unit (CDU), which replaces the fully-connected layers of
standard CNNs. Our experiments demonstrate the benefits of our new
architecture, which outperform the first-order CNNs, while relying on up to 90%
fewer parameters.

本文提出一种新颖的卷积神经网络（CNN）类，它利用二阶统计量，可以形成协方差描述符单元（CDU），可以替代标准 CNN 的全连接层，并在参数少至 90％的情况下优于一阶 CNN。