Recent research has explored the implementation of privacy-preserving deep
neural networks solely using fully homomorphic encryption. However, its
practicality has been limited because of prolonged inference times. When using
a pre-trained model without retraining, a major factor contributing to these
prolonged inference times is the high-degree polynomial approximation of
activation functions such as the ReLU function. The high-degree approximation
consumes a substantial amount of homomorphic computational resources, resulting
in slower inference. Unlike the previous works approximating activation
functions uniformly and conservatively, this paper presents a \emph{layerwise}
degree optimization of activation functions to aggressively reduce the
inference time while maintaining classification accuracy by taking into account
the characteristics of each layer. Instead of the minimax approximation
commonly used in state-of-the-art private inference models, we employ the
weighted least squares approximation method with the input distributions of
activation functions. Then, we obtain the layerwise optimized degrees for
activation functions through the \emph{dynamic programming} algorithm,
considering how each layer's approximation error affects the classification
accuracy of the deep neural network. Furthermore, we propose modulating the
ciphertext moduli-chain layerwise to reduce the inference time. By these
proposed layerwise optimization methods, we can reduce inference times for the
ResNet-20 model and the ResNet-32 model by 3.44 times and 3.16 times,
respectively, in comparison to the prior implementations employing uniform
degree polynomials and a consistent ciphertext modulus.

通过对激活函数进行逐层优化并调整密文模量链，本文提出了一种降低推理时间的隐私保护深度神经网络实现方法，可以将 ResNet-20 模型和 ResNet-32 模型的推理时间分别缩短 3.44 倍和 3.16 倍。