Transformers are the dominant architecture for sequence modeling, but there
is growing interest in models that use a fixed-size latent state that does not
depend on the sequence length, which we refer to as "generalized state space
models" (GSSMs). In this paper we show that while GSSMs are promising in terms
of inference-time efficiency, they are limited compared to transformer models
on tasks that require copying from the input context. We start with a
theoretical analysis of the simple task of string copying and prove that a two
layer transformer can copy strings of exponential length while GSSMs are
fundamentally limited by their fixed-size latent state. Empirically, we find
that transformers outperform GSSMs in terms of efficiency and generalization on
synthetic tasks that require copying the context. Finally, we evaluate
pretrained large language models and find that transformer models dramatically
outperform state space models at copying and retrieving information from
context. Taken together, these results suggest a fundamental gap between
transformers and GSSMs on tasks of practical interest.

在这篇论文中，我们研究了使用不依赖于序列长度的固定大小的潜在状态的模型（我们将其称为 “广义状态空间模型”（GSSMs））与 transformer 模型在需要从输入上下文中进行复制的任务上的性能差异。我们从对简单的字符串复制任务的理论分析开始，并证明了一个二层 transformer 可以复制指数长度的字符串，而 GSSMs 因为固定大小的潜在状态而受到限制。在实证研究中，我们发现 transformers 在需要复制上下文的合成任务上效果优于 GSSMs，无论是在效率还是在泛化方面。最后，我们评估了预训练的大型语言模型，并发现 transformer 模型在复制和检索上下文信息的任务上远远胜过状态空间模型。综合这些结果表明，在实际任务中，transformers 与 GSSMs 之间存在着根本的差距。