When trained on language data, do transformers learn some arbitrary
computation that utilizes the full capacity of the architecture or do they
learn a simpler, tree-like computation, hypothesized to underlie compositional
meaning systems like human languages? There is an apparent tension between
compositional accounts of human language understanding, which are based on a
restricted bottom-up computational process, and the enormous success of neural
models like transformers, which can route information arbitrarily between
different parts of their input. One possibility is that these models, while
extremely flexible in principle, in practice learn to interpret language
hierarchically, ultimately building sentence representations close to those
predictable by a bottom-up, tree-structured model. To evaluate this
possibility, we describe an unsupervised and parameter-free method to
\emph{functionally project} the behavior of any transformer into the space of
tree-structured networks. Given an input sentence, we produce a binary tree
that approximates the transformer's representation-building process and a score
that captures how "tree-like" the transformer's behavior is on the input. While
calculation of this score does not require training any additional models, it
provably upper-bounds the fit between a transformer and any tree-structured
approximation. Using this method, we show that transformers for three different
tasks become more tree-like over the course of training, in some cases
unsupervisedly recovering the same trees as supervised parsers. These trees, in
turn, are predictive of model behavior, with more tree-like models generalizing
better on tests of compositional generalization.

研究表明，当进行语言数据训练时，transformers 是学习一种简单的基于树状结构的计算过程，可以解释像人类语言这样的复合意义系统，这种过程没有必要使用其完整架构的全部容量。通过无监督和无参数的方法，可以在任何 transformer 和树状结构之间建立功能投影，而三项不同的任务实验表明，在一些情况下，可无监督地恢复相同的树状结构，这些树状结构有助于模型的行为改进。