We survey results on neural network expressivity described in "On the
Expressive Power of Deep Neural Networks". The paper motivates and develops
three natural measures of expressiveness, which all display an exponential
dependence on the depth of the network. In fact, all of these measures are
related to a fourth quantity, trajectory length. This quantity grows
exponentially in the depth of the network, and is responsible for the depth
sensitivity observed. These results translate to consequences for networks
during and after training. They suggest that parameters earlier in a network
have greater influence on its expressive power -- in particular, given a layer,
its influence on expressivity is determined by the remaining depth of the
network after that layer. This is verified with experiments on MNIST and
CIFAR-10. We also explore the effect of training on the input-output map, and
find that it trades off between the stability and expressivity.

本文研究神经网络的表达能力，提出了三种衡量表达能力的自然量，并探讨了神经网络层数与表达能力之间的关系，特别地找出了深度敏感性所在，同时分析了训练和输入 - 输出映射等因素对表达能力的影响。

深度神经网络表达能力调查

Survey of Expressivity in Deep Neural Networks

We propose a new approach to the problem of neural network expressivity,
which seeks to characterize how structural properties of a neural network
family affect the functions it is able to compute. Our approach is based on an
interrelated set of measures of expressivity, unified by the novel notion of
trajectory length, which measures how the output of a network changes as the
input sweeps along a one-dimensional path. Our findings can be summarized as
follows:
(1) The complexity of the computed function grows exponentially with depth.
(2) All weights are not equal: trained networks are more sensitive to their
lower (initial) layer weights.
(3) Regularizing on trajectory length (trajectory regularization) is a
simpler alternative to batch normalization, with the same performance.

本文提出了一种新的神经网络表达性问题的方法，其中基于轨迹长度的一维路径上的输出是一种新颖的表达形式。实验得出：（1）计算的函数复杂度随深度指数增长；（2）所有权重不同，加上轨迹正则化是批标准化的一个更简单的选择，但表现相同。