We advance the recently proposed neuro-symbolic Differentiable Tree Machine,
which learns tree operations using a combination of transformers and Tensor
Product Representations. We investigate the architecture and propose two key
components. We first remove a series of different transformer layers that are
used in every step by introducing a mixture of experts. This results in a
Differentiable Tree Experts model with a constant number of parameters for any
arbitrary number of steps in the computation, compared to the previous method
in the Differentiable Tree Machine with a linear growth. Given this flexibility
in the number of steps, we additionally propose a new termination algorithm to
provide the model the power to choose how many steps to make automatically. The
resulting Terminating Differentiable Tree Experts model sluggishly learns to
predict the number of steps without an oracle. It can do so while maintaining
the learning capabilities of the model, converging to the optimal amount of
steps.

我们提出了一种基于神经符号不可微分树模型的改进方法，通过使用变压器和张量乘积表示学习树操作，并引入了专家混合物和终止算法，以灵活选择执行步骤和预测最优步数。

终止可微分树专家

Terminating Differentiable Tree Experts

Recurrent neural networks (RNNs) can learn continuous vector representations
of symbolic structures such as sequences and sentences; these representations
often exhibit linear regularities (analogies). Such regularities motivate our
hypothesis that RNNs that show such regularities implicitly compile symbolic
structures into tensor product representations (TPRs; Smolensky, 1990), which
additively combine tensor products of vectors representing roles (e.g.,
sequence positions) and vectors representing fillers (e.g., particular words).
To test this hypothesis, we introduce Tensor Product Decomposition Networks
(TPDNs), which use TPRs to approximate existing vector representations. We
demonstrate using synthetic data that TPDNs can successfully approximate linear
and tree-based RNN autoencoder representations, suggesting that these
representations exhibit interpretable compositional structure; we explore the
settings that lead RNNs to induce such structure-sensitive representations. By
contrast, further TPDN experiments show that the representations of four models
trained to encode naturally-occurring sentences can be largely approximated
with a bag of words, with only marginal improvements from more sophisticated
structures. We conclude that TPDNs provide a powerful method for interpreting
vector representations, and that standard RNNs can induce compositional
sequence representations that are remarkably well approximated by TPRs; at the
same time, existing training tasks for sentence representation learning may not
be sufficient for inducing robust structural representations.

使用 Tensor Product Decomposition Networks（TPDNs）来近似现有向量表示，证明 RNNs 可以通过 Tensor Product Representations（TPRs）引导序列表示。

循环神经网络隐含地实现张量积表示

RNNs Implicitly Implement Tensor Product Representations

We combine Recurrent Neural Networks with Tensor Product Representations to
learn combinatorial representations of sequential data. This improves symbolic
interpretation and systematic generalisation. Our architecture is trained
end-to-end through gradient descent on a variety of simple natural language
reasoning tasks, significantly outperforming the latest state-of-the-art models
in single-task and all-tasks settings. We also augment a subset of the data
such that training and test data exhibit large systematic differences and show
that our approach generalises better than the previous state-of-the-art.

本文采用递归神经网络与张量积表示相结合的方法，学习顺序数据的组合表示并在多个简单的自然语言推理任务中取得显著优于最新先进模型的表现，同时证明该方法的泛化性能优于现有最新先进技术。