Activation functions play a significant role in neural network design by
enabling non-linearity. The choice of activation function was previously shown
to influence the properties of the resulting loss landscape. Understanding the
relationship between activation functions and loss landscape properties is
important for neural architecture and training algorithm design. This study
empirically investigates neural network loss landscapes associated with
hyperbolic tangent, rectified linear unit, and exponential linear unit
activation functions. Rectified linear unit is shown to yield the most convex
loss landscape, and exponential linear unit is shown to yield the least flat
loss landscape, and to exhibit superior generalisation performance. The
presence of wide and narrow valleys in the loss landscape is established for
all activation functions, and the narrow valleys are shown to correlate with
saturated neurons and implicitly regularised network configurations.

此研究对与神经网络有关的损失景观进行了实证调查，发现修正线性单元产生最凸的损失景观，而指数线性单元产生最不平坦的损失景观，且表现出优越的泛化性能。

神经网络激活函数的经验损失景观分析

Empirical Loss Landscape Analysis of Neural Network Activation Functions

Object recognition is an important task for improving the ability of visual
systems to perform complex scene understanding. Recently, the Exponential
Linear Unit (ELU) has been proposed as a key component for managing bias shift
in Convolutional Neural Networks (CNNs), but defines a parameter that must be
set by hand. In this paper, we propose learning a parameterization of ELU in
order to learn the proper activation shape at each layer in the CNNs. Our
results on the MNIST, CIFAR-10/100 and ImageNet datasets using the NiN,
Overfeat, All-CNN and ResNet networks indicate that our proposed Parametric ELU
(PELU) has better performances than the non-parametric ELU. We have observed as
much as a 7.28% relative error improvement on ImageNet with the NiN network,
with only 0.0003% parameter increase. Our visual examination of the non-linear
behaviors adopted by Vgg using PELU shows that the network took advantage of
the added flexibility by learning different activations at different layers.

提出了一种参数化 ELU 激活函数的方法以学习 CNN 中每层的恰当激活形状，实验结果表明，参数化的 ELU 在目标识别任务中相对误差较小，性能更优。

深度卷积神经网络参数化的指数线性单元

Parametric Exponential Linear Unit for Deep Convolutional Neural  Networks

Very deep convolutional neural networks introduced new problems like
vanishing gradient and degradation. The recent successful contributions towards
solving these problems are Residual and Highway Networks. These networks
introduce skip connections that allow the information (from the input or those
learned in earlier layers) to flow more into the deeper layers. These very deep
models have lead to a considerable decrease in test errors, on benchmarks like
ImageNet and COCO. In this paper, we propose the use of exponential linear unit
instead of the combination of ReLU and Batch Normalization in Residual
Networks. We show that this not only speeds up learning in Residual Networks
but also improves the accuracy as the depth increases. It improves the test
error on almost all data sets, like CIFAR-10 and CIFAR-100

本文旨在指出在深度卷积神经网络中引入跳过连接从而解决挥发的渐变问题的新方案，并提出将指数线性单元用于残差网络，以显著减小测试误差。

带有指数线性单元的深度残差网络

Deep Residual Networks with Exponential Linear Unit

We introduce the "exponential linear unit" (ELU) which speeds up learning in
deep neural networks and leads to higher classification accuracies. Like
rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs
(PReLUs), ELUs alleviate the vanishing gradient problem via the identity for
positive values. However, ELUs have improved learning characteristics compared
to the units with other activation functions. In contrast to ReLUs, ELUs have
negative values which allows them to push mean unit activations closer to zero
like batch normalization but with lower computational complexity. Mean shifts
toward zero speed up learning by bringing the normal gradient closer to the
unit natural gradient because of a reduced bias shift effect. While LReLUs and
PReLUs have negative values, too, they do not ensure a noise-robust
deactivation state. ELUs saturate to a negative value with smaller inputs and
thereby decrease the forward propagated variation and information. Therefore,
ELUs code the degree of presence of particular phenomena in the input, while
they do not quantitatively model the degree of their absence. In experiments,
ELUs lead not only to faster learning, but also to significantly better
generalization performance than ReLUs and LReLUs on networks with more than 5
layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with
batch normalization while batch normalization does not improve ELU networks.
ELU networks are among the top 10 reported CIFAR-10 results and yield the best
published result on CIFAR-100, without resorting to multi-view evaluation or
model averaging. On ImageNet, ELU networks considerably speed up learning
compared to a ReLU network with the same architecture, obtaining less than 10%
classification error for a single crop, single model network.

引入了 “指数线性单元”（ELU），可以提高深度神经网络的学习速度和分类准确性，相较于其他激活函数，ELU 具有更好的学习特征和更低的计算复杂度，能够在输入数据中编码特定现象的存在程度，并在实验中取得比其他激活函数更快的学习和更好的泛化表现，是目前 CIFAR-100 最好的分类结果之一。