We present a unified representation of the most popular neural network
activation functions. Adopting Mittag-Leffler functions of fractional calculus,
we propose a flexible and compact functional form that is able to interpolate
between various activation functions and mitigate common problems in training
neural networks such as vanishing and exploding gradients. The presented gated
representation extends the scope of fixed-shape activation functions to their
adaptive counterparts whose shape can be learnt from the training data. The
derivatives of the proposed functional form can also be expressed in terms of
Mittag-Leffler functions making it a suitable candidate for gradient-based
backpropagation algorithms. By training LeNet-5 neural network on MNIST and
CIFAR-10 datasets, we demonstrate that adopting a unified gated representation
of activation functions offers a promising and affordable alternative to
individual built-in implementations of activation functions in conventional
machine learning frameworks.

我们提出了一种统一的神经网络激活函数表示形式，采用分数阶微积分的 Mittag-Leffler 函数，该形式能够插值不同的激活函数并减轻神经网络训练中普遍存在的梯度消失和梯度爆炸等问题。使用 Lenet-5 神经网络在 MNIST 和 CIFAR-10 数据集上训练，我们证明采用统一的门控表示法是传统机器学习框架中内置的激活函数实现的一种有前途且经济的替代方案。