Understanding the power of depth in feed-forward neural networks is an
ongoing challenge in the field of deep learning theory. While current works
account for the importance of depth for the expressive power of
neural-networks, it remains an open question whether these benefits are
exploited during a gradient-based optimization process. In this work we explore
the relation between expressivity properties of deep networks and the ability
to train them efficiently using gradient-based algorithms. We give a depth
separation argument for distributions with fractal structure, showing that they
can be expressed efficiently by deep networks, but not with shallow ones. These
distributions have a natural coarse-to-fine structure, and we show that the
balance between the coarse and fine details has a crucial effect on whether the
optimization process is likely to succeed. We prove that when the distribution
is concentrated on the fine details, gradient-based algorithms are likely to
fail. Using this result we prove that, at least in some distributions, the
success of learning deep networks depends on whether the distribution can be
well approximated by shallower networks, and we conjecture that this property
holds in general.

该论文研究了深层神经网络在梯度下降最优化过程中利用深度的表达能力，证明了具有分形结构的分布可以被深层网络有效地表达，而浅层网络无法表达。论文还探讨了粗细篮子之间的平衡如何影响深度神经网络的优化过程，并推断了学习深度神经网络是否成功取决于分布是否可以被浅层网络很好地逼近的结论。