Recent advances in deep learning have relied heavily on the use of large
Transformers due to their ability to learn at scale. However, the core building
block of Transformers, the attention operator, exhibits quadratic cost in
sequence length, limiting the amount of context accessible. Existing
subquadratic methods based on low-rank and sparse approximations need to be
combined with dense attention layers to match Transformers, indicating a gap in
capability. In this work, we propose Hyena, a subquadratic drop-in replacement
for attention constructed by interleaving implicitly parametrized long
convolutions and data-controlled gating. In recall and reasoning tasks on
sequences of thousands to hundreds of thousands of tokens, Hyena improves
accuracy by more than 50 points over operators relying on state-spaces and
other implicit and explicit methods, matching attention-based models. We set a
new state-of-the-art for dense-attention-free architectures on language
modeling in standard datasets (WikiText103 and The Pile), reaching Transformer
quality with a 20% reduction in training compute required at sequence length
2K. Hyena operators are twice as fast as highly optimized attention at sequence
length 8K, and 100x faster at sequence length 64K.

本文介绍了 Hyena，一种亚二次的 attention 替代方法，它通过交错的启发式参数化长卷积和数据控制开关的方式进行构建，能够有效地解决 Transformers 中 quadratic cost 的问题，并且在大规模自然语言处理任务中实现了 Transformer 模型一致的精度，同时减少了 20% 的训练计算资源。