We introduce a new construction of embeddings of arbitrary recursive data
structures into high dimensional vectors. These embeddings provide an
interpretable model for the latent state vectors of transformers. We
demonstrate that these embeddings can be decoded to the original data structure
when the embedding dimension is sufficiently large. This decoding algorithm has
a natural implementation as a transformer. We also show that these embedding
vectors can be manipulated directly to perform computations on the underlying
data without decoding. As an example we present an algorithm that constructs
the embedded parse tree of an embedded token sequence using only vector
operations in embedding space.

我们介绍了一种将任意递归数据结构嵌入到高维向量中的新构建方法，这些嵌入为变压器的潜在状态向量提供了可解释的模型。我们展示了当嵌入维度足够大时，这些嵌入可以解码为原始数据结构。这个解码算法在变压器中有一个自然的实现。我们还展示了这些嵌入向量可以直接操作底层数据执行计算而无需解码。例如，我们提出了一个算法，仅使用嵌入空间的向量操作构造嵌入的标记序列的嵌入句法树。

Banach-Tarski 嵌入与变换器

Banach-Tarski Embeddings and Transformers

This paper studies the dynamic generator model for spatial-temporal processes
such as dynamic textures and action sequences in video data. In this model,
each time frame of the video sequence is generated by a generator model, which
is a non-linear transformation of a latent state vector, where the non-linear
transformation is parametrized by a top-down neural network. The sequence of
latent state vectors follows a non-linear auto-regressive model, where the
state vector of the next frame is a non-linear transformation of the state
vector of the current frame as well as an independent noise vector that
provides randomness in the transition. The non-linear transformation of this
transition model can be parametrized by a feedforward neural network. We show
that this model can be learned by an alternating back-propagation through time
algorithm that iteratively samples the noise vectors and updates the parameters
in the transition model and the generator model. We show that our training
method can learn realistic models for dynamic textures and action patterns.

本研究研究了动态生成模型的空间 - 时间过程，如视频数据中的动态纹理和动作序列，模型通过一个生成模型产生视频序列中的每个时间帧，生成模型是潜在状态向量的非线性转换，并由自回归模型控制。