Recurrent Neural Networks (RNNs) are very successful at solving challenging
problems with sequential data. However, this observed efficiency is not yet
entirely explained by theory. It is known that a certain class of
multiplicative RNNs enjoys the property of depth efficiency --- a shallow
network of exponentially large width is necessary to realize the same score
function as computed by such an RNN. Such networks, however, are not very often
applied to real life tasks. In this work, we attempt to reduce the gap between
theory and practice by extending the theoretical analysis to RNNs which employ
various nonlinearities, such as Rectified Linear Unit (ReLU), and show that
they also benefit from properties of universality and depth efficiency. Our
theoretical results are verified by a series of extensive computational
experiments.

本文研究了使用不同非线性激活函数的循环神经网络（RNNs）的理论效率，表明它们也具有普适性和深度效率的特性，并通过计算实验证实了这一理论结果。

适用于循环神经网络的广义张量模型

Generalized Tensor Models for Recurrent Neural Networks

A key attribute that drives the unprecedented success of modern Recurrent
Neural Networks (RNNs) on learning tasks which involve sequential data, is
their ability to model intricate long-term temporal dependencies. However, a
well established measure of RNNs long-term memory capacity is lacking, and thus
formal understanding of the effect of depth on their ability to correlate data
throughout time is limited. Specifically, existing depth efficiency results on
convolutional networks do not suffice in order to account for the success of
deep RNNs on data of varying lengths. In order to address this, we introduce a
measure of the network's ability to support information flow across time,
referred to as the Start-End separation rank, which reflects the distance of
the function realized by the recurrent network from modeling no dependency
between the beginning and end of the input sequence. We prove that deep
recurrent networks support Start-End separation ranks which are combinatorially
higher than those supported by their shallow counterparts. Thus, we establish
that depth brings forth an overwhelming advantage in the ability of recurrent
networks to model long-term dependencies, and provide an exemplar of
quantifying this key attribute which may be readily extended to other RNN
architectures of interest, e.g. variants of LSTM networks. We obtain our
results by considering a class of recurrent networks referred to as Recurrent
Arithmetic Circuits, which merge the hidden state with the input via the
Multiplicative Integration operation, and empirically demonstrate the discussed
phenomena on common RNNs. Finally, we employ the tool of quantum Tensor
Networks to gain additional graphic insight regarding the complexity brought
forth by depth in recurrent networks.

本文通过引入一种称为 “起始 - 终止分离等级” 的度量方式，证明了深度循环网络相较于其浅层对应版本在建模长时依赖方面具有压倒性的优势，并通过量子张量网络的工具获取了更多的图形洞见。