We investigate the training and generalization errors of overparameterized
neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU)
functions. More specifically, we carefully upper bound both the convergence
rate of the training error and the generalization error of such NNs and
investigate the dependence of these bounds on the Leaky ReLU parameter,
$\alpha$. We show that $\alpha =-1$, which corresponds to the absolute value
activation function, is optimal for the training error bound. Furthermore, in
special settings, it is also optimal for the generalization error bound.
Numerical experiments empirically support the practical choices guided by the
theory.

我们研究了具有广泛的漏斗修复线性统一函数的过参数神经网络的训练和泛化误差。我们对此类神经网络的训练误差的收敛速率和泛化误差进行了上界估计，并研究了这些上界与漏斗修复线性统一参数 alpha 的依赖关系。我们证明了 alpha 等于 - 1，对应于绝对值激活函数，是训练误差上界的最优选择。此外，在特定的设置下，它也是泛化误差上界的最优选择。数值实验在实践中支持了由理论指导的实际选择。

泄漏 ReLU 在超参数化网络的训练和泛化中的影响

The effect of Leaky ReLUs on the training and generalization of  overparameterized networks

Activation functions (AFs) are crucial components of deep neural networks
(DNNs), having a significant impact on their performance. An activation
function in a DNN is typically a smooth, nonlinear function that transforms an
input signal into an output signal for the subsequent layer. In this paper, we
propose the Parametric Leaky Tanh (PLTanh), a novel hybrid activation function
designed to combine the strengths of both the Tanh and Leaky ReLU (LReLU)
activation functions. PLTanh is differentiable at all points and addresses the
'dying ReLU' problem by ensuring a non-zero gradient for negative inputs,
consistent with the behavior of LReLU. By integrating the unique advantages of
these two diverse activation functions, PLTanh facilitates the learning of more
intricate nonlinear relationships within the network. This paper presents an
empirical evaluation of PLTanh against established activation functions, namely
ReLU, LReLU, and ALReLU utilizing five diverse datasets.

本文在深度神经网络中提出了 Parametric Leaky Tanh（PLTanh）作为一种新的混合激活函数，结合了 Tanh 和 Leaky ReLU 激活函数的优势，通过保证负输入的梯度非零，解决了 'dying ReLU' 问题，并通过与 ReLU、LReLU 和 ALReLU 在五个不同数据集上的实证评估，展示了 PLTanh 的优势。

参数泄漏双曲正切：一种用于深度学习的新型混合激活函数

Parametric Leaky Tanh: A New Hybrid Activation Function for Deep  Learning

In this paper we investigate the performance of different types of rectified
activation functions in convolutional neural network: standard rectified linear
unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified
linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU).
We evaluate these activation function on standard image classification task.
Our experiments suggest that incorporating a non-zero slope for negative part
in rectified activation units could consistently improve the results. Thus our
findings are negative on the common belief that sparsity is the key of good
performance in ReLU. Moreover, on small scale dataset, using deterministic
negative slope or learning it are both prone to overfitting. They are not as
effective as using their randomized counterpart. By using RReLU, we achieved
75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.

本文通过评估不同类型的修正线性单元 rectified activation functions（包括：标准修正线性单元（ReLU），泄漏修正线性单元（Leaky ReLU），参数修正线性单元（PReLU）以及随机泄漏修正线性单元（RReLU））在图像分类任务中的表现，结论表明，对修正激活单元中的负部分引入非零斜率可以始终改善结果，从而推翻了稀疏性是 ReLU 良好性能的关键的常见信念。另外，在小规模数据集上，使用确定性的负斜率或学习固定斜率都容易过拟合，使用随机斜率则更为有效。通过使用 RReLU，我们在 CIFAR-100 测试集上实现了 75.68％的准确度（无多次测试或集合）。