The driving force behind deep networks is their ability to compactly
represent rich classes of functions. The primary notion for formally reasoning
about this phenomenon is expressive efficiency, which refers to a situation
where one network must grow unfeasibly large in order to realize (or
approximate) functions of another. To date, expressive efficiency analyses
focused on the architectural feature of depth, showing that deep networks are
representationally superior to shallow ones. In this paper we study the
expressive efficiency brought forth by connectivity, motivated by the
observation that modern networks interconnect their layers in elaborate ways.
We focus on dilated convolutional networks, a family of deep models delivering
state of the art performance in sequence processing tasks. By introducing and
analyzing the concept of mixed tensor decompositions, we prove that
interconnecting dilated convolutional networks can lead to expressive
efficiency. In particular, we show that even a single connection between
intermediate layers can already lead to an almost quadratic gap, which in
large-scale settings typically makes the difference between a model that is
practical and one that is not. Empirical evaluation demonstrates how the
expressive efficiency of connectivity, similarly to that of depth, translates
into gains in accuracy. This leads us to believe that expressive efficiency may
serve a key role in the development of new tools for deep network design.

本文研究通过连接层数的增加来增强深度网络的表达效率，应用于序列处理任务中的卷积神经网络，证明了深度和连接都能提高表现效果，并将这一发现应用于深度网络设计。

混合张量分解增强扩张卷积网络

Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions

Expressive efficiency refers to the relation between two architectures A and
B, whereby any function realized by B could be replicated by A, but there
exists functions realized by A, which cannot be replicated by B unless its size
grows significantly larger. For example, it is known that deep networks are
exponentially efficient with respect to shallow networks, in the sense that a
shallow network must grow exponentially large in order to approximate the
functions represented by a deep network of polynomial size. In this work, we
extend the study of expressive efficiency to the attribute of network
connectivity and in particular to the effect of "overlaps" in the convolutional
process, i.e., when the stride of the convolution is smaller than its filter
size (receptive field). To theoretically analyze this aspect of network's
design, we focus on a well-established surrogate for ConvNets called
Convolutional Arithmetic Circuits (ConvACs), and then demonstrate empirically
that our results hold for standard ConvNets as well. Specifically, our analysis
shows that having overlapping local receptive fields, and more broadly denser
connectivity, results in an exponential increase in the expressive capacity of
neural networks. Moreover, while denser connectivity can increase the
expressive capacity, we show that the most common types of modern architectures
already exhibit exponential increase in expressivity, without relying on
fully-connected layers.

研究表明，神经网络的表达能力会随着局部感受域的重叠和更密集的连接而呈指数增长，最近的一些现代体系结构已经展示了指数级的表达能力提升，无需依赖于完全连接的层。