Recent works (e.g., (Li and Arora, 2020)) suggest that the use of popular
normalization schemes (including Batch Normalization) in today's deep learning
can move it far from a traditional optimization viewpoint, e.g., use of
exponentially increasing learning rates. The current paper highlights other
ways in which behavior of normalized nets departs from traditional viewpoints,
and then initiates a formal framework for studying their mathematics via
suitable adaptation of the conventional framework namely, modeling SGD-induced
training trajectory via a suitable stochastic differential equation (SDE) with
a noise term that captures gradient noise. This yields: (a) A new ' intrinsic
learning rate' parameter that is the product of the normal learning rate and
weight decay factor. Analysis of the SDE shows how the effective speed of
learning varies and equilibrates over time under the control of intrinsic LR.
(b) A challenge -- via theory and experiments -- to popular belief that good
generalization requires large learning rates at the start of training. (c) New
experiments, backed by mathematical intuition, suggesting the number of steps
to equilibrium (in function space) scales as the inverse of the intrinsic
learning rate, as opposed to the exponential time convergence bound implied by
SDE analysis. We name it the Fast Equilibrium Conjecture and suggest it holds
the key to why Batch Normalization is effective.

本研究通过建立适当的随机微分方程、分析训练轨迹和实验，提出了深度学习中标准化网络的行为远离传统视角的其他方面，并发现了基于 “内部学习率” 的参数，控制了学习的速度和均衡，在深度学习优化和泛化方面具有新的挑战和实验结论，这些结果对于深度学习中的规范化技术具有丰富的理论和实践意义。