Private computation of nonlinear functions, such as Rectified Linear Units
(ReLUs) and max-pooling operations, in deep neural networks (DNNs) poses
significant challenges in terms of storage, bandwidth, and time consumption. To
address these challenges, there has been a growing interest in utilizing
privacy-preserving techniques that leverage polynomial activation functions and
kernelized convolutions as alternatives to traditional ReLUs. However, these
alternative approaches often suffer from a trade-off between achieving faster
private inference (PI) and sacrificing model accuracy. In particular, when
applied to much deeper networks, these methods encounter training
instabilities, leading to issues like exploding gradients (resulting in NaNs)
or suboptimal approximations. In this study, we focus on PolyKervNets, a
technique known for offering improved dynamic approximations in smaller
networks but still facing instabilities in larger and more complex networks.
Our primary objective is to empirically explore optimization-based training
recipes to enhance the performance of PolyKervNets in larger networks. By doing
so, we aim to potentially eliminate the need for traditional nonlinear
activation functions, thereby advancing the state-of-the-art in
privacy-preserving deep neural network architectures. Code can be found on
GitHub at: https://github.com/tolusophy/PolyKervNets/

通过优化训练方法，提高 PolyKervNets 在大型网络中的性能，可能消除传统非线性激活函数的需求，推动隐私保护深度神经网络架构的最新发展。