In a recent paper [B. Li, S. Tang and H. Yu, arXiv:1903.05858], it was shown
that deep neural networks built with rectified power units (RePU) can give
better approximation for sufficient smooth functions than those with rectified
linear units, by converting polynomial approximation given in power series into
deep neural networks with optimal complexity and no approximation error.
However, in practice, power series are not easy to compute. In this paper, we
propose a new and more stable way to construct deep RePU neural networks based
on Chebyshev polynomial approximations. By using a hierarchical structure of
Chebyshev polynomial approximation in frequency domain, we build efficient and
stable deep neural network constructions. In theory, ChebNets and the deep RePU
nets based on Power series have the same upper error bounds for general
function approximations. But numerically, ChebNets are much more stable.
Numerical results show that the constructed ChebNets can be further trained and
obtain much better results than those obtained by training deep RePU nets
constructed basing on power series.

本文探讨了如何构建基于 Chebyshev 多项式逼近的深度 RePU 神经网络，不仅理论上和利用幂级数逼近构建的 RePU 网络具有相同的上限误差界限，而且通过实验发现，ChebNets 比使用幂级数逼近的 RePU 网络更加稳定。

ChebNet: 基于切比雪夫逼近的修正幂单元深度神经网络的高效稳定构建

ChebNet: Efficient and Stable Constructions of Deep Neural Networks with  Rectified Power Units using Chebyshev Approximations

Deep neural networks with rectified linear units (ReLU) are getting more and
more popular due to their universal representation power and successful
applications. Some theoretical progress regarding the approximation power of
deep ReLU network for functions in Sobolev space and Korobov space have
recently been made by [D. Yarotsky, Neural Network, 94:103-114, 2017] and [H.
Montanelli and Q. Du, SIAM J Math. Data Sci., 1:78-92, 2019], etc. In this
paper, we show that deep networks with rectified power units (RePU) can give
better approximations for smooth functions than deep ReLU networks. Our
analysis bases on classical polynomial approximation theory and some efficient
algorithms proposed in this paper to convert polynomials into deep RePU
networks of optimal size with no approximation error. Comparing to the results
on ReLU networks, the sizes of RePU networks required to approximate functions
in Sobolev space and Korobov space with an error tolerance $\varepsilon$, by
our constructive proofs, are in general
$\mathcal{O}(\log\frac{1}{\varepsilon})$ times smaller than the sizes of
corresponding ReLU networks constructed in most of the existing literature.
Comparing to the classical results of Mhaskar [Mhaskar, Adv. Comput. Math.
1:61-80, 1993], our constructions use less number of activation functions and
numerically more stable, they can be served as good initials of deep RePU
networks and further trained to break the limit of linear approximation theory.
The functions represented by RePU networks are smooth functions, so they
naturally fit in the places where derivatives are involved in the loss
function.

本文研究使用深层神经网络进行平滑函数的逼近，提出了基于经典多项式逼近理论的优化算法与框架，并发现使用修正功率单元的网络对于逼近平滑函数比使用修正线性单元的网络具有更好的表现，网络的大小也更为紧凑，适用于损失函数中涉及求导的场合。