We explore the phase diagram of approximation rates for deep neural networks. The phase diagram describes theoretically optimal accuracy-complexity relations and their qualitative properties. Our contribution is three-fold. First, we generalize the existing result on the existence of deep discontinuous phase in ReLU networks to functional classes of arbitrary positive smoothness, and identify the boundary between the feasible and infeasible rates. Second, we demonstrate that standard fully-connected architectures of a fixed width independent of smoothness can adapt to smoothness and achieve almost optimal rates. Finally, we discuss how the phase diagram can change in the case of non-ReLU activation functions. In particular, we prove that using both sine and ReLU activations theoretically leads to very fast, nearly exponential approximation rates, thanks to the emerging capability of the network to implement efficient lookup operations.

该论文探讨了深度神经网络的近似速率相图，并证明了几个新的理论结果，具体而言，将ReLU网络中深度不连续相的现有结果推广到了任意正光滑的函数类，并确定了可行和不可行速率之间的界限；此外，我们表明所有具有分段多项式激活函数的网络具有相同的相图；接下来，我们证明了具有固定宽度且独立于光滑度的标准全连接架构可以适应光滑度并实现几乎最佳速率；最后，我们考虑了具有周期性激活（“深度傅立叶展开”）的深层网络，并证明它们具有非常快的指数近似速率，这要归功于网络实现高效查找操作的新能力。

深度神经网络逼近率的相图