We examine the characteristic activation values of individual ReLU units in
neural networks. We refer to the corresponding set for such characteristic
activation values in the input space as the characteristic activation set of a
ReLU unit. We draw an explicit connection between the characteristic activation
set and learned features in ReLU networks. This connection leads to new
insights into why various neural network normalization techniques used in
modern deep learning architectures regularize and stabilize SGD optimization.
Utilizing these insights, we propose a geometric approach to parameterize ReLU
networks for improved feature learning. We empirically verify its usefulness
with less carefully chosen initialization schemes and larger learning rates. We
report improved optimization stability, faster convergence speed, and better
generalization performance.

本文研究神经网络中 individual ReLU units 的 characteristic activation values，提出 ReLU unit 的 characteristic activation set 与 ReLU networks 中的 learned features 之间的联系，由此推导出新的洞见 —— 哪些现代深度学习中的规范化技术可用于正则化和稳定化 SGD 优化，最后提出几何方法用于对 ReLU 网络进行参数化以提高特征学习。实验证明该方法可通过简单调整初始化策略和更大的学习速率来提高优化稳定性、收敛速度和泛化性能。

神经特征激活值分析以提高 ReLU 网络特征学习

Neural Characteristic Activation Value Analysis for Improved ReLU  Network Feature Learning

Hyperparameter tuning is one of the most time-consuming workloads in deep
learning. State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam,
reduce this labor by adaptively tuning an individual learning rate for each
variable. Recently researchers have shown renewed interest in simpler methods
like momentum SGD as they may yield better test metrics. Motivated by this
trend, we ask: can simple adaptive methods based on SGD perform as well or
better? We revisit the momentum SGD algorithm and show that hand-tuning a
single learning rate and momentum makes it competitive with Adam. We then
analyze its robustness to learning rate misspecification and objective
curvature variation. Based on these insights, we design YellowFin, an automatic
tuner for momentum and learning rate in SGD. YellowFin optionally uses a
negative-feedback loop to compensate for the momentum dynamics in asynchronous
settings on the fly. We empirically show that YellowFin can converge in fewer
iterations than Adam on ResNets and LSTMs for image recognition, language
modeling and constituency parsing, with a speedup of up to 3.28x in synchronous
and up to 2.69x in asynchronous settings.

YellowFin 是一种基于 SGD 优化的自动调参方法，通过手动调整学习率和动量，可以达到和 Adam 优化器相当的效果，同时在异步训练环境下，引入负反馈回路可以进一步提高收敛速度。