When trained on language data, do transformers learn some arbitrary
computation that utilizes the full capacity of the architecture or do they
learn a simpler, tree-like computation, hypothesized to underlie compositional
meaning systems like human languages? There is an apparent tension between
compositional accounts of human language understanding, which are based on a
restricted bottom-up computational process, and the enormous success of neural
models like transformers, which can route information arbitrarily between
different parts of their input. One possibility is that these models, while
extremely flexible in principle, in practice learn to interpret language
hierarchically, ultimately building sentence representations close to those
predictable by a bottom-up, tree-structured model. To evaluate this
possibility, we describe an unsupervised and parameter-free method to
\emph{functionally project} the behavior of any transformer into the space of
tree-structured networks. Given an input sentence, we produce a binary tree
that approximates the transformer's representation-building process and a score
that captures how "tree-like" the transformer's behavior is on the input. While
calculation of this score does not require training any additional models, it
provably upper-bounds the fit between a transformer and any tree-structured
approximation. Using this method, we show that transformers for three different
tasks become more tree-like over the course of training, in some cases
unsupervisedly recovering the same trees as supervised parsers. These trees, in
turn, are predictive of model behavior, with more tree-like models generalizing
better on tests of compositional generalization.

研究表明，当进行语言数据训练时，transformers 是学习一种简单的基于树状结构的计算过程，可以解释像人类语言这样的复合意义系统，这种过程没有必要使用其完整架构的全部容量。通过无监督和无参数的方法，可以在任何 transformer 和树状结构之间建立功能投影，而三项不同的任务实验表明，在一些情况下，可无监督地恢复相同的树状结构，这些树状结构有助于模型的行为改进。

利用树状投影表征 Transformer 中的固有组合性

Characterizing Intrinsic Compositionality in Transformers with Tree Projections

We present the multiplicative recurrent neural network as a general model for
compositional meaning in language, and evaluate it on the task of fine-grained
sentiment analysis. We establish a connection to the previously investigated
matrix-space models for compositionality, and show they are special cases of
the multiplicative recurrent net. Our experiments show that these models
perform comparably or better than Elman-type additive recurrent neural networks
and outperform matrix-space models on a standard fine-grained sentiment
analysis corpus. Furthermore, they yield comparable results to structural deep
models on the recently published Stanford Sentiment Treebank without the need
for generating parse trees.

本文通过提出一个多重循环神经网络的新模型，其可以广泛应用于语言的组合式含义，还可用于微观情感分析任务。研究者通过这种模型建立了与之前研究的矩阵空间模型建立了联系，并证明它们是该乘法网络的特例。 该研究发现其表现比 Elman 型加性循环神经网络要好，在标准的微观情感分析语料库上优于矩阵空间模型，并且不需要生成句子的语法树，并且在最近发布的斯坦福情感树库上产生与结构深层模型相当的结果。

采用乘法循环神经网络模拟组合性

Modeling Compositionality with Multiplicative Recurrent Neural Networks

Modelling compositional meaning for sentences using empirical distributional
methods has been a challenge for computational linguists. We implement the
abstract categorical model of Coecke et al. (arXiv:1003.4394v1 [cs.CL]) using
data from the BNC and evaluate it. The implementation is based on unsupervised
learning of matrices for relational words and applying them to the vectors of
their arguments. The evaluation is based on the word disambiguation task
developed by Mitchell and Lapata (2008) for intransitive sentences, and on a
similar new experiment designed for transitive sentences. Our model matches the
results of its competitors in the first experiment, and betters them in the
second. The general improvement in results with increase in syntactic
complexity showcases the compositional power of our model.

利用经验分配法对句子进行组合意义建模一直是计算语言学家面临的挑战。我们利用 BNC 数据实现了 Coecke 等人的抽象范畴模型，并进行了评估。实现基于无监督学习的关系词矩阵，应用于其参数的向量。我们的模型在失歧任务上拥有比竞争对手更优异的表现，而随着句法复杂度的增加，结果的普遍改善展示了我们模型的组合能力。