In this paper we investigate the performance of different types of rectified
activation functions in convolutional neural network: standard rectified linear
unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified
linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU).
We evaluate these activation function on standard image classification task.
Our experiments suggest that incorporating a non-zero slope for negative part
in rectified activation units could consistently improve the results. Thus our
findings are negative on the common belief that sparsity is the key of good
performance in ReLU. Moreover, on small scale dataset, using deterministic
negative slope or learning it are both prone to overfitting. They are not as
effective as using their randomized counterpart. By using RReLU, we achieved
75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble.

本文通过评估不同类型的修正线性单元 rectified activation functions（包括：标准修正线性单元（ReLU），泄漏修正线性单元（Leaky ReLU），参数修正线性单元（PReLU）以及随机泄漏修正线性单元（RReLU））在图像分类任务中的表现，结论表明，对修正激活单元中的负部分引入非零斜率可以始终改善结果，从而推翻了稀疏性是 ReLU 良好性能的关键的常见信念。另外，在小规模数据集上，使用确定性的负斜率或学习固定斜率都容易过拟合，使用随机斜率则更为有效。通过使用 RReLU，我们在 CIFAR-100 测试集上实现了 75.68％的准确度（无多次测试或集合）。