Real-world analog systems intrinsically suffer from noise that can impede
model convergence and accuracy on a variety of deep learning models. We
demonstrate that differentiable activations like GELU and SiLU enable robust
propagation of gradients which help to mitigate analog quantization error that
is ubiquitous to all analog systems. We perform analysis and training of
convolutional, linear, and transformer networks in the presence of quantized
noise. Here, we are able to demonstrate that continuously differentiable
activation functions are significantly more noise resilient over conventional
rectified activations. As in the case of ReLU, the error in gradients are 100x
higher than those in GELU near zero. Our findings provide guidance for
selecting appropriate activations to realize performant and reliable hardware
implementations across several machine learning domains such as computer
vision, signal processing, and beyond.

通过不可微的激活函数如 GELU 和 SiLU，我们能够在存在量化噪声的情况下，实现对卷积、线性和 Transformer 网络的分析和训练，从而提供实现高性能和可靠硬件的适当激活函数选择。