Linear transformers aim to reduce the quadratic space-time complexity of
vanilla transformers. However, they usually suffer from degraded performances
on various tasks and corpus. In this paper, we examine existing kernel-based
linear transformers and identify two key issues that lead to such performance
gaps: 1) unbounded gradients in the attention computation adversely impact the
convergence of linear transformer models; 2) attention dilution which trivially
distributes attention scores over long sequences while neglecting neighbouring
structures. To address these issues, we first identify that the scaling of
attention matrices is the devil in unbounded gradients, which turns out
unnecessary in linear attention as we show theoretically and empirically. To
this end, we propose a new linear attention that replaces the scaling operation
with a normalization to stabilize gradients. For the issue of attention
dilution, we leverage a diagonal attention to confine attention to only
neighbouring tokens in early layers. Benefiting from the stable gradients and
improved attention, our new linear transformer model, transNormer, demonstrates
superior performance on text classification and language modeling tasks, as
well as on the challenging Long-Range Arena benchmark, surpassing vanilla
transformer and existing linear variants by a clear margin while being
significantly more space-time efficient. The code is available at
this https URL .

本文提出了一种新的线性变压器模型，称为 transNormer，以解决现有线性变压器的性能问题，通过稳定梯度和改善注意力的方式，在文本分类和语言建模任务以及 Long-Range Arena 基准测试中表现出更优异的性能，同时更加高效。

线性变换中的魔鬼

The Devil in Linear Transformer

A proper parametrization of state transition matrices of linear state-space
models (SSMs) followed by standard nonlinearities enables them to efficiently
learn representations from sequential data, establishing the state-of-the-art
on a large series of long-range sequence modeling benchmarks. In this paper, we
show that we can improve further when the structural SSM such as S4 is given by
a linear liquid time-constant (LTC) state-space model. LTC neural networks are
causal continuous-time neural networks with an input-dependent state transition
module, which makes them learn to adapt to incoming inputs at inference. We
show that by using a diagonal plus low-rank decomposition of the state
transition matrix introduced in S4, and a few simplifications, the LTC-based
structural state-space model, dubbed Liquid-S4, achieves the new
state-of-the-art generalization across sequence modeling tasks with long-term
dependencies such as image, text, audio, and medical time-series, with an
average performance of 87.32% on the Long-Range Arena benchmark. On the full
raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with
a 30% reduction in parameter counts compared to S4. The additional gain in
performance is the direct result of the Liquid-S4's kernel structure that takes
into account the similarities of the input sequence samples during training and
inference.

本文介绍了一种新的状态空间结构 Liquid-S4 ，它可以通过对状态转移矩阵的对角线及低秩分解进一步提高序列建模任务的表现，达到了 Long-Range Arena 基准测试的最新水平，并在 Speech Command 识别数据集上实现了 96.78% 的精度。