We study the capabilities of the transformer architecture with varying depth.
Specifically, we designed a novel set of sequence learning tasks to
systematically evaluate and comprehend how the depth of transformer affects its
ability to perform memorization, reasoning, generalization, and contextual
generalization. We show a transformer with only one attention layer can excel
in memorization but falls short in other tasks. Then, we show that exhibiting
reasoning and generalization ability requires the transformer to have at least
two attention layers, while context generalization ability may necessitate
three attention layers. Additionally, we identify a class of simple operations
that a single attention layer can execute, and show that the complex tasks can
be approached as the combinations of these simple operations and thus can be
resolved by stacking multiple attention layers. This sheds light on studying
more practical and complex tasks beyond our design. Numerical experiments
corroborate our theoretical findings.

我们研究了具有不同深度的 transformer 架构的能力，通过设计了一套新的序列学习任务系统地评估和理解深度对 transformer 在记忆、推理、泛化和上下文泛化方面的影响。我们发现只有一个注意力层的 transformer 在记忆方面表现出色，但在其他任务方面表现不足。我们还发现，要表现出推理和泛化能力，transformer 至少需要两个注意力层，而上下文泛化能力可能需要三个注意力层。此外，我们还确定了单个注意力层可以执行的一类简单操作，并且表明复杂任务可以通过这些简单操作的组合以及多个注意力层的叠加来解决。这为研究超越我们设计的更实际和复杂的任务提供了启示。数值实验证实了我们的理论发现。

变换器在不同深度下能学到什么？对序列学习任务的案例研究

What Can Transformer Learn with Varying Depth? Case Studies on Sequence  Learning Tasks

The resonate-and-fire (RF) neuron, introduced over two decades ago, is a
simple, efficient, yet biologically plausible spiking neuron model, which can
extract frequency patterns within the time domain due to its resonating
membrane dynamics. However, previous RF formulations suffer from intrinsic
shortcomings that limit effective learning and prevent exploiting the
principled advantage of RF neurons. Here, we introduce the balanced RF (BRF)
neuron, which alleviates some of the intrinsic limitations of vanilla RF
neurons and demonstrates its effectiveness within recurrent spiking neural
networks (RSNNs) on various sequence learning tasks. We show that networks of
BRF neurons achieve overall higher task performance, produce only a fraction of
the spikes, and require significantly fewer parameters as compared to modern
RSNNs. Moreover, BRF-RSNN consistently provide much faster and more stable
training convergence, even when bridging many hundreds of time steps during
backpropagation through time (BPTT). These results underscore that our BRF-RSNN
is a strong candidate for future large-scale RSNN architectures, further lines
of research in SNN methodology, and more efficient hardware implementations.

平衡 RF 神经元在循环脉冲神经网络中证明了其在各种序列学习任务中的有效性，其任务性能更高、产生的脉冲更少、所需参数更少，并且在训练过程中具有更快且更稳定的收敛速度。

平衡的共振 - 放火神经元

Balanced Resonate-and-Fire Neurons

While the multi-branch architecture is one of the key ingredients to the
success of computer vision tasks, it has not been well investigated in natural
language processing, especially sequence learning tasks. In this work, we
propose a simple yet effective variant of Transformer called multi-branch
attentive Transformer (briefly, MAT), where the attention layer is the average
of multiple branches and each branch is an independent multi-head attention
layer. We leverage two training techniques to regularize the training:
drop-branch, which randomly drops individual branches during training, and
proximal initialization, which uses a pre-trained Transformer model to
initialize multiple branches. Experiments on machine translation, code
generation and natural language understanding demonstrate that such a simple
variant of Transformer brings significant improvements. Our code is available
at https://github.com/HA-Transformer.

本研究提出了一种名为多分支注意力 Transformer（MAT）的变体，通过平均多个分支的注意力层，并使用两种训练技术 —— 随机丢弃分支和相似初始化，对机器翻译、代码生成和自然语言理解等任务进行了实验，并取得了显著改善。