State-space models are a low-complexity alternative to transformers for
encoding long sequences and capturing long-term dependencies. We propose
LOCOST: an encoder-decoder architecture based on state-space models for
conditional text generation with long context inputs. With a computational
complexity of $O(L \log L)$, this architecture can handle significantly longer
sequences than state-of-the-art models that are based on sparse attention
patterns. We evaluate our model on a series of long document abstractive
summarization tasks. The model reaches a performance level that is 93-96%
comparable to the top-performing sparse transformers of the same size while
saving up to 50% memory during training and up to 87% during inference.
Additionally, LOCOST effectively handles input texts exceeding 600K tokens at
inference time, setting new state-of-the-art results on full-book summarization
and opening new perspectives for long input processing.

LOCOST 是基于状态空间模型的编码器 - 解码器结构，用于带有长上下文输入的条件文本生成，具有低复杂度的计算和能够处理超长序列的能力。在长篇摘要任务上评估模型，其性能可与同等规模的顶级稀疏转换器相媲美，同时在训练过程中节省高达 50% 的内存，在推理过程中节省高达 87% 的内存。此外，LOCOST 有效地处理超过 600K 个标记的输入文本，在全书摘要任务上达到了新的最先进水平，并为长输入处理开展了新的研究视角。

LOCOST: 面向长文档提要生成的状态空间模型

LOCOST: State-Space Models for Long Document Abstractive Summarization

Transformers provide a class of expressive architectures that are extremely
effective for sequence modeling. However, the key limitation of transformers is
their quadratic memory and time complexity $\mathcal{O}(L^2)$ with respect to
the sequence length in attention layers, which restricts application in
extremely long sequences. Most existing approaches leverage sparsity or
low-rank assumptions in the attention matrix to reduce cost, but sacrifice
expressiveness. Instead, we propose Combiner, which provides full attention
capability in each attention head while maintaining low computation and memory
complexity. The key idea is to treat the self-attention mechanism as a
conditional expectation over embeddings at each location, and approximate the
conditional distribution with a structured factorization. Each location can
attend to all other locations, either via direct attention, or through indirect
attention to abstractions, which are again conditional expectations of
embeddings from corresponding local regions. We show that most sparse attention
patterns used in existing sparse transformers are able to inspire the design of
such factorization for full attention, resulting in the same sub-quadratic cost
($\mathcal{O}(L\log(L))$ or $\mathcal{O}(L\sqrt{L})$). Combiner is a drop-in
replacement for attention layers in existing transformers and can be easily
implemented in common frameworks. An experimental evaluation on both
autoregressive and bidirectional sequence tasks demonstrates the effectiveness
of this approach, yielding state-of-the-art results on several image and text
modeling tasks.

提出了一种名为 Combiner 的替代 transformers 中的 attention layers 的方法，其通过利用结构化分解的条件分布来近似自我关注机制，实现了每个关注头的完全关注能力，同时保持低计算和内存复杂性。在自回归和双向序列任务上的实验评估表明，该方法有效性高，可以产生多个图像和文本建模任务的最新技术结果。