This article concerns the expressive power of depth in deep feed-forward
neural nets with ReLU activations. Specifically, we answer the following
question: for a fixed $d_{in}\geq 1,$ what is the minimal width $w$ so that
neural nets with ReLU activations, input dimension $d_{in}$, hidden layer
widths at most $w,$ and arbitrary depth can approximate any continuous,
real-valued function of $d_{in}$ variables arbitrarily well? It turns out that
this minimal width is exactly equal to $d_{in}+1.$ That is, if all the hidden
layer widths are bounded by $d_{in}$, then even in the infinite depth limit,
ReLU nets can only express a very limited class of functions, and, on the other
hand, any continuous function on the $d_{in}$-dimensional unit cube can be
approximated to arbitrary precision by ReLU nets in which all hidden layers
have width exactly $d_{in}+1.$ Our construction in fact shows that any
continuous function $f:[0,1]^{d_{in}}\to\mathbb R^{d_{out}}$ can be
approximated by a net of width $d_{in}+d_{out}$. We obtain quantitative depth
estimates for such an approximation in terms of the modulus of continuity of
$f$.

研究了 ReLU 激活的深度前馈神经网络的表达能力问题，得出结论：使用该网络结构可以以任意精度逼近任意 $d_{in}$ 维的连续实值函数，需要的最小宽度为 $d_{in}+1$，而一般深度和宽度都受限时，则只能表达并逼近有限的函数集。最后提出任何连续函数都可以通过宽度为 $d_{in}+d_{out}$ 的网络逼近，且该逼近的确切程度与函数的连续性有关。