The performance of deep network learning strongly depends on the choice of
the non-linear activation function associated with each neuron. However,
deciding on the best activation is non-trivial, and the choice depends on the
architecture, hyper-parameters, and even on the dataset. Typically these
activations are fixed by hand before training. Here, we demonstrate how to
eliminate the reliance on first picking fixed activation functions by using
flexible parametric rational functions instead. The resulting Pad\'e Activation
Units (PAUs) can both approximate common activation functions and also learn
new ones while providing compact representations. Our empirical evidence shows
that end-to-end learning deep networks with PAUs can increase the predictive
performance. Moreover, PAUs pave the way to approximations with provable
robustness. this https URL

本文中介绍了一种新的非线性激活函数 ——Padé 激活单元 (PAUs)，能够代替传统手动选择的固定激活函数，增加深度神经网络的预测性能和可靠性。