Recent advances in attention-free sequence models rely on convolutions as
alternatives to the attention operator at the core of Transformers. In
particular, long convolution sequence models have achieved state-of-the-art
performance in many domains, but incur a significant cost during
auto-regressive inference workloads -- naively requiring a full pass (or
caching of activations) over the input sequence for each generated token --
similarly to attention-based models. In this paper, we seek to enable $\mathcal
O(1)$ compute and memory cost per token in any pre-trained long convolution
architecture to reduce memory footprint and increase throughput during
generation. Concretely, our methods consist in extracting low-dimensional
linear state-space models from each convolution layer, building upon rational
interpolation and model-order reduction techniques. We further introduce
architectural improvements to convolution-based layers such as Hyena: by
weight-tying the filters across channels into heads, we achieve higher
pre-training quality and reduce the number of filters to be distilled. The
resulting model achieves 10x higher throughput than Transformers and 1.5x
higher than Hyena at 1.3B parameters, without any loss in quality after
distillation.

通过提取低维线性状态空间模型以及改进卷积层的结构，本文实现了每个标记的 O (1) 计算和内存开销，从而降低内存占用并提高生成过程的吞吐量，而无需牺牲质量。