Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.

本研究从两个方面研究了基于修正线性单元（rectifier）的神经网络在图像分类方面的应用：首先，本文提出了一种广义化的参数修正线性单元（PReLU）来改进模型拟合；其次，本文提出了一种健壮的初始化方法来考虑线性单元的非线性，使我们能够直接从头开始训练极深层的网络。通过使用我们提出的 PReLU 网络，我们在 ImageNet 2012 数据集上实现了 4.94% 的 top-5 测试错误率，相对于ILSVRC 2014 的冠军GoogLeNet的6.66％，这是一个26％的相对改进。根据我们的知识，我们的结果是首次在这个视觉识别挑战中超越了人类水平表现（5.1％，Russakovsky等）。

深入探索整流器：在ImageNet分类上超越人类水平表现