The time complexity of the standard attention mechanism in a transformer
scales quadratically with the length of the sequence. We introduce a method to
reduce this to linear scaling with time, based on defining attention via latent
vectors. The method is readily usable as a drop-in replacement for the standard
attention mechanism. Our "Latte Transformer" model can be implemented for both
bidirectional and unidirectional tasks, with the causal version allowing a
recurrent implementation which is memory and time-efficient during inference of
language generation tasks. Whilst next token prediction scales linearly with
the sequence length for a standard transformer, a Latte Transformer requires
constant time to compute the next token. The empirical performance of our
method is comparable to standard attention, yet allows scaling to context
windows much larger than practical in standard attention.

在传统的 transformer 模型中，标准的 attention 机制的时间复杂度随着序列的长度呈二次方增长。本研究提出了一种基于潜在向量定义注意力的方法，将时间复杂度降低为随序列长度线性增长。我们的 “Latte Transformer” 模型可以在双向和单向任务中使用，通过因果版本可以实现在推理过程中进行语言生成任务的记忆和时间高效的循环实现。与标准 transformer 相比，标准的下一个记号预测的时间复杂度与序列长度成线性关系，而 Latte Transformer 只需常数时间计算下一个记号。我们方法的实证表现与标准 attention 相当，但允许在标准 attention 不可行的背景窗口范围内进行扩展。