Transformer architectures have been widely adopted in foundation models. Due
to their high inference costs, there is renewed interest in exploring the
potential of efficient recurrent architectures (RNNs). In this paper, we
analyze the differences in the representational capabilities of Transformers
and RNNs across several tasks of practical relevance, including index lookup,
nearest neighbor, recognizing bounded Dyck languages, and string equality. For
the tasks considered, our results show separations based on the size of the
model required for different architectures. For example, we show that a
one-layer Transformer of logarithmic width can perform index lookup, whereas an
RNN requires a hidden state of linear size. Conversely, while constant-size
RNNs can recognize bounded Dyck languages, we show that one-layer Transformers
require a linear size for this task. Furthermore, we show that two-layer
Transformers of logarithmic size can perform decision tasks such as string
equality or disjointness, whereas both one-layer Transformers and recurrent
models require linear size for these tasks. We also show that a log-size
two-layer Transformer can implement the nearest neighbor algorithm in its
forward pass; on the other hand recurrent models require linear size. Our
constructions are based on the existence of $N$ nearly orthogonal vectors in
$O(\log N)$ dimensional space and our lower bounds are based on reductions from
communication complexity problems. We supplement our theoretical results with
experiments that highlight the differences in the performance of these
architectures on practical-size sequences.

通过实验和理论分析比较了 Transformer 和 RNN 在不同任务上的表征能力、模型大小以及性能差异。

Transformer 和循环体系结构在表示能力上的区别

Separations in the Representational Capabilities of Transformers and  Recurrent Architectures

Large Language Models (LLMs) do not differentially represent numbers, which
are pervasive in text. In contrast, neuroscience research has identified
distinct neural representations for numbers and words. In this work, we
investigate how well popular LLMs capture the magnitudes of numbers (e.g., that
$4 < 5$) from a behavioral lens. Prior research on the representational
capabilities of LLMs evaluates whether they show human-level performance, for
instance, high overall accuracy on standard benchmarks. Here, we ask a
different question, one inspired by cognitive science: How closely do the
number representations of LLMscorrespond to those of human language users, who
typically demonstrate the distance, size, and ratio effects? We depend on a
linking hypothesis to map the similarities among the model embeddings of number
words and digits to human response times. The results reveal surprisingly
human-like representations across language models of different architectures,
despite the absence of the neural circuitry that directly supports these
representations in the human brain. This research shows the utility of
understanding LLMs using behavioral benchmarks and points the way to future
work on the number of representations of LLMs and their cognitive plausibility.

本文研究大型语言模型（LLMs）是否能够准确地表示数字，并发现尽管缺少直接支持这些表示的神经电路，不同结构的 LLMs 中存在着类似于人类语言使用者的数字表示，这证明了使用行为基准理解 LLMs 的实用性，并为今后关于 LLMs 数字表示和它们的认知合理性的研究指出了方向。