Research into the expressive power of neural networks typically considers real parameters and operations without rounding error. In this work, we study universal approximation property of quantized networks under discrete fixed-point parameters and fixed-point operations that may incur errors due to rounding. We first provide a necessary condition and a sufficient condition on fixed-point arithmetic and activation functions for universal approximation of quantized networks. Then, we show that various popular activation functions satisfy our sufficient condition, e.g., Sigmoid, ReLU, ELU, SoftPlus, SiLU, Mish, and GELU. In other words, networks using those activation functions are capable of universal approximation. We further show that our necessary condition and sufficient condition coincide under a mild condition on activation functions: e.g., for an activation function $\sigma$, there exists a fixed-point number $x$ such that $\sigma(x)=0$. Namely, we find a necessary and sufficient condition for a large class of activation functions. We lastly show that even quantized networks using binary weights in $\{-1,1\}$ can also universally approximate for practical activation functions.

本研究解决了量化神经网络在固定点算术下的普适逼近性质的问题，填补了当前通常考虑实数参数和无舍入误差的研究空白。通过提供针对固定点算术和激活函数的必要条件和充分条件，证明了多种流行的激活函数可以实现普适逼近，特别是在对激活函数施加轻微条件后，得到了一类激活函数的必要与充分条件，最终表明即使是使用二值权重的量化网络也能够进行普适逼近。

固定点算术下量化神经网络的表达能力研究