The choice of step-size used in Stochastic Gradient Descent (SGD)
optimization is empirically selected in most training procedures. Moreover, the
use of scheduled learning techniques such as Step-Decaying, Cyclical-Learning,
and Warmup to tune the step-size requires extensive practical
experience--offering limited insight into how the parameters update--and is not
consistent across applications. This work attempts to answer a question of
interest to both researchers and practitioners, namely \textit{"how much
knowledge is gained in iterative training of deep neural networks?"} Answering
this question introduces two useful metrics derived from the singular values of
the low-rank factorization of convolution layers in deep neural networks. We
introduce the notions of \textit{"knowledge gain"} and \textit{"mapping
condition"} and propose a new algorithm called Adaptive Scheduling (AdaS) that
utilizes these derived metrics to adapt the SGD learning rate proportionally to
the rate of change in knowledge gain over successive iterations.
Experimentation reveals that, using the derived metrics, AdaS exhibits: (a)
faster convergence and superior generalization over existing adaptive learning
methods; and (b) lack of dependence on a validation set to determine when to
stop training. Code is available at
https://github.com/mahdihosseini/AdaS.

本文提出了一种名为 “自适应调度” 的算法，该算法利用奇异值得出的两个指标 “知识增益” 和 “映射条件”，并将 SGD 学习率与知识增益的变化率成比例地调整，实验表明该算法具有更快的收敛速度和更好的泛化性能。