We propose two new criteria to understand the advantage of deepening neural
networks. It is important to know the expressivity of functions computable by
deep neural networks in order to understand the advantage of deepening neural
networks. Unless deep neural networks have enough expressivity, they cannot
have good performance even though learning is successful. In this situation,
the proposed criteria contribute to understanding the advantage of deepening
neural networks since they can evaluate the expressivity independently from the
efficiency of learning. The first criterion shows the approximation accuracy of
deep neural networks to the target function. This criterion has the background
that the goal of deep learning is approximating the target function by deep
neural networks. The second criterion shows the property of linear regions of
functions computable by deep neural networks. This criterion has the background
that deep neural networks whose activation functions are piecewise linear are
also piecewise linear. Furthermore, by the two criteria, we show that to
increase layers is more effective than to increase units at each layer on
improving the expressivity of deep neural networks.

该论文提出了两个新标准，以评估深度神经网络的表达能力和计算函数，并同时比较增加层数和增加每层神经元数量对于提高模型表达能力的效果，从而增进对深度学习的理解。