BriefGPT.xyz
Oct, 2019
深度学习指数学习率调度
An Exponential Learning Rate Schedule for Deep Learning
HTML
PDF
Zhiyuan Li, Sanjeev Arora
TL;DR
通过对BN的权重衰减及动量模型的应用,本文发现深度学习算法能够成功应用于具有指数增长学习速率的训练方式,证明了这种训练方式在各种标准结构中具有优秀的表现,并给出了数学解释和实例验证。
Abstract
Intriguing empirical evidence exists that
deep learning
can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenonmay be due to
batch normalization
or BN(Ioffe & Sze
→