Existing depth separation results for constant-depth networks essentially
show that certain radial functions in $\mathbb{R}^d$, which can be easily
approximated with depth $3$ networks, cannot be approximated by depth $2$
networks, even up to constant accuracy, unless their size is exponential in
$d$. However, the functions used to demonstrate this are rapidly oscillating,
with a Lipschitz parameter scaling polynomially with the dimension $d$ (or
equivalently, by scaling the function, the hardness result applies to
$\mathcal{O}(1)$-Lipschitz functions only when the target accuracy $\epsilon$
is at most $\text{poly}(1/d)$). In this paper, we study whether such depth
separations might still hold in the natural setting of
$\mathcal{O}(1)$-Lipschitz radial functions, when $\epsilon$ does not scale
with $d$. Perhaps surprisingly, we show that the answer is negative: In
contrast to the intuition suggested by previous work, it \emph{is} possible to
approximate $\mathcal{O}(1)$-Lipschitz radial functions with depth $2$, size
$\text{poly}(d)$ networks, for every constant $\epsilon$. We complement it by
showing that approximating such functions is also possible with depth $2$, size
$\text{poly}(1/\epsilon)$ networks, for every constant $d$. Finally, we show
that it is not possible to have polynomial dependence in both $d,1/\epsilon$
simultaneously. Overall, our results indicate that in order to show depth
separations for expressing $\mathcal{O}(1)$-Lipschitz functions with constant
accuracy -- if at all possible -- one would need fundamentally different
techniques than existing ones in the literature.

考虑常数精度下 Lipschitz 参数不随维度 d 变化时的 O (1) Lipschitz 径向函数问题，发现相比之前的研究，不存在深度为 2、规模为 poly (d) 的神经网络可近似表示该类函数，但当深度和规模限制任意一个为 poly (1/epsilon) 时，函数可被近似表示，两者的多项式依赖程度不能同时为多项式，表明要证明相应深度分离结果需要全新技术支持。

神经网络中的深度分离：实际分离的是什么？

Depth Separations in Neural Networks: What is Actually Being Separated?

Based on the tree architecture, the objective of this paper is to design deep
neural networks with two or more hidden layers (called deep nets) for
realization of radial functions so as to enable rotational invariance for
near-optimal function approximation in an arbitrarily high dimensional
Euclidian space. It is shown that deep nets have much better performance than
shallow nets (with only one hidden layer) in terms of approximation accuracy
and learning capabilities. In particular, for learning radial functions, it is
shown that near-optimal rate can be achieved by deep nets but not by shallow
nets. Our results illustrate the necessity of depth in neural network design
for realization of rotation-invariance target functions.

本研究基于树形结构探讨如何设计深度神经网络用于实现径向函数，以实现在任意高维欧几里得空间内旋转不变性的近乎最优函数逼近。结果显示，深度网络在逼近精度和学习能力方面远优于仅具有一个隐藏层的浅层神经网络，并证明了对于学习径向函数，深度网络可以实现近乎最优的学习速率，而浅层网络却不能。因此，这项研究说明深度在神经网络设计中的必要性，以实现旋转不变的目标函数。