This paper introduces a significantly better class of activation functions
than the almost universally used ReLU like and Sigmoidal class of activation
functions. Two new activation functions referred to as the Cone and
Parabolic-Cone that differ drastically from popular activation functions and
significantly outperform these on the CIFAR-10 and Imagenette benchmmarks are
proposed. The cone activation functions are positive only on a finite interval
and are strictly negative except at the end-points of the interval, where they
become zero. Thus the set of inputs that produce a positive output for a neuron
with cone activation functions is a hyperstrip and not a half-space as is the
usual case. Since a hyper strip is the region between two parallel
hyper-planes, it allows neurons to more finely divide the input feature space
into positive and negative classes than with infinitely wide half-spaces. In
particular the XOR function can be learn by a single neuron with cone-like
activation functions. Both the cone and parabolic-cone activation functions are
shown to achieve higher accuracies with significantly fewer neurons on
benchmarks. The results presented in this paper indicate that many nonlinear
real-world datasets may be separated with fewer hyperstrips than half-spaces.
The Cone and Parabolic-Cone activation functions have larger derivatives than
ReLU and are shown to significantly speedup training.

介绍了两种新的激活函数，Cone 和 Parabolic-Cone，相较于常用的 ReLU 和 Sigmoidal 类激活函数，在 CIFAR-10 和 Imagenette 两个基准测试中明显表现更好。这些激活函数在有限区间内为正，且在区间的端点处变为零，使得神经元可以更精细地将输入特征空间划分为正类和负类，通过较少的超带学习 XOR 函数，并且在基准测试中达到更高的准确率。此研究表明，在许多非线性真实世界的数据集中，相较于半空间，较少的超带可以实现数据的分离，并且 Cone 和 Parabolic-Cone 激活函数具有更大的导数，加速了训练过程。