We study the necessary and sufficient complexity of ReLU neural networks---in terms of depth and number of weights---which is required for approximating classifier functions in an $L^2$-sense. As a model class, we consider the set $\mathcal{E}^\beta (\mathbb R^d)$ of possibly discontinuous piecewise $C^\beta$ functions $f : [-1/2, 1/2]^d \to \mathbb R$, where the different "smooth regions" of $f$ are separated by $C^\beta$ hypersurfaces. For given dimension $d \geq 2$, regularity $\beta > 0$, and accuracy $\varepsilon > 0$, we construct artificial neural networks with ReLU activation function that approximate functions from $\mathcal{E}^\beta(\mathbb R^d)$ up to an $L^2$ error of $\varepsilon$. The constructed networks have a fixed number of layers, depending only on $d$ and $\beta$ and they have $\mathcal{O}(\varepsilon^{-2(d-1)/\beta})$ many non-zero weights, which we prove to be optimal. For the proof of optimality, we establish a lower bound on the description complexity of the class $\mathcal{E}^\beta (\mathbb R^d)$. By showing that a family of approximating neural networks gives rise to an encoder for $\mathcal{E}^\beta (\mathbb R^d)$, we then prove that one cannot approximate a general function $f \in \mathcal{E}^\beta (\mathbb R^d)$ using neural networks that are less complex than those produced by our construction. In addition to the optimality in terms of the number of weights, we show that in order to achieve this optimal approximation rate, one needs ReLU networks of a certain minimal depth. Precisely, for piecewise $C^\beta(\mathbb R^d)$ functions, this minimal depth is given---up to a multiplicative constant---by $\beta/d$. Up to a log factor, our constructed networks match this bound. This partly explains the benefits of depth for ReLU networks by showing that deep networks are necessary to achieve efficient approximation of (piecewise) smooth functions.

研究了在$L^2$意义下逼近分类器函数所需的ReLU神经网络的深度和权重数量，构造了一类具有固定层数的人工神经网络，使用ReLU激活函数逼近可允许不连续的分段$C^β$函数，权重数量为$O(ε^{-(2(d-1))/β})$，并证明这是最优的。此外，为了实现最优逼近率，需要具有一定深度的ReLU网络。最后，分析了在高维空间中使用特征映射和分类器函数逼近的情况。

使用深度 ReLU 神经网络对分段光滑函数进行最优逼近