Transformers with linear attention allow for efficient parallel training but
can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden
states, thus enjoying linear (with respect to output length) inference
complexity. Recent works such as RetNet (Sun et al., 2023) and TransNormerLLM
(Qin et al., 2023a) observe that adding a global decay term to the additive RNN
update rule greatly improves performance, sometimes outperforming standard
Transformers with softmax attention when trained at scale. In this work we show
that adding a data-dependent gating mechanism further improves performance. We
derive a parallel form of this gated linear attention layer that enables
efficient training. However, a straightforward, numerically stable
implementation of this parallel form requires generalized matrix
multiplications in log-space for numerical stability, and thus cannot take
advantage of tensor cores on modern GPUs which are optimized for standard
matrix multiplications. We develop a hardware-efficient version of the parallel
form that can still make use of tensor cores through block-parallel
computations over sequence chunks. Experiments on moderate-scale language
modeling (340M-parameter models trained on 15B tokens, 1.3B-parameter models
trained on 100B tokens) show that gated linear attention (GLA) Transformers
perform competitively against a strong LLaMA-architecture Transformer baseline
(Touvron et al., 2023) as well as Mamba (Gu & Dao, 2023), a recently introduced
state-space model with a data-dependent state transition mechanism. For
training speed, our Triton-based implementation performs comparably to
CUDA-optimized FlashAttention-2 (Dao, 2023) under the regular 2048 training
length setting, while outperforming FlashAttention-2 when training on longer
sequences beyond 4096.

通过引入数据依赖的门控机制，我们开发了一种硬件高效的并行形式，使得门控线性注意力（GLA）Transformers 在适度规模的语言建模中表现竞争力，并在训练速度上与基于 CUDA 优化的 FlashAttention-2 相媲美。

硬件高效训练的门控线性注意力变换器

Gated Linear Attention Transformers with Hardware-Efficient Training

Can we automatically design a Convolutional Network (ConvNet) with the
highest image classification accuracy under the runtime constraint of a mobile
device? Neural architecture search (NAS) has revolutionized the design of
hardware-efficient ConvNets by automating this process. However, the NAS
problem remains challenging due to the combinatorially large design space,
causing a significant searching time (at least 200 GPU-hours). To alleviate
this complexity, we propose Single-Path NAS, a novel differentiable NAS method
for designing hardware-efficient ConvNets in less than 4 hours. Our
contributions are as follows: 1. Single-path search space: Compared to previous
differentiable NAS methods, Single-Path NAS uses one single-path
over-parameterized ConvNet to encode all architectural decisions with shared
convolutional kernel parameters, hence drastically decreasing the number of
trainable parameters and the search cost down to few epochs. 2.
Hardware-efficient ImageNet classification: Single-Path NAS achieves 74.96%
top-1 accuracy on ImageNet with 79ms latency on a Pixel 1 phone, which is
state-of-the-art accuracy compared to NAS methods with similar constraints
(<80ms). 3. NAS efficiency: Single-Path NAS search cost is only 8 epochs (30
TPU-hours), which is up to 5,000x faster compared to prior work. 4.
Reproducibility: Unlike all recent mobile-efficient NAS methods which only
release pretrained models, we open-source our entire codebase at:
this https URL

本研究提出了 Single-Path NAS，这是一种新的可微分 NAS 方法，用于在少于 4 小时的时间内设计硬件高效的 ConvNets，并在移动设备下实现最高图像分类准确性。