We propose a novel tensor network language model based on the simplest tensor network (i.e., tensor trains), called `Tensor Train Language Model' (TTLM). TTLM represents sentences in an exponential space constructed by the tensor product of words, but computing the probabilities of sentences in a low-dimensional fashion. We demonstrate that the architectures of Second-order RNNs, Recurrent Arithmetic Circuits (RACs), and Multiplicative Integration RNNs are, essentially, special cases of TTLM. Experimental evaluations on real language modeling tasks show that the proposed variants of TTLM (i.e., TTLM-Large and TTLM-Tiny) outperform the vanilla Recurrent Neural Networks (RNNs) with low-scale of hidden units. (The code is available at https://github.com/shuishen112/tensortrainlm.)

我们提出了一种基于最简单的张量网络（即张量列）的新型张量网络语言模型，称为'Tensor Train Language Model'（TTLM）。通过将单词的张量积构建成指数空间，TTLM表示句子，但以低维度方式计算句子的概率。我们证明了第二阶RNN、递归算术电路（RACs）和乘积积分RNN的架构本质上是TTLM的特殊情况。对真实语言建模任务的实验评估表明，TTLM的各种变体（即TTLM-Large和TTLM-Tiny）优于具有低规模隐层单元的传统递归神经网络（RNNs）。

使用张量列进行语言建模