In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart.

本文通过评估不同类型的修正线性单元rectified activation functions（包括：标准修正线性单元（ReLU），泄漏修正线性单元（Leaky ReLU），参数修正线性单元（PReLU）以及随机泄漏修正线性单元（RReLU））在图像分类任务中的表现，结论表明，对修正激活单元中的负部分引入非零斜率可以始终改善结果，从而推翻了稀疏性是ReLU良好性能的关键的常见信念。另外，在小规模数据集上，使用确定性的负斜率或学习固定斜率都容易过拟合，使用随机斜率则更为有效。通过使用RReLU，我们在CIFAR-100测试集上实现了75.68％的准确度（无多次测试或集合）。

卷积网络中修正的激活函数的实证评估