We present a strikingly simple proof that two rules are sufficient to
automate gradient descent: 1) don't increase the stepsize too fast and 2) don't
overstep the local curvature. No need for functional values, no line search, no
information about the function except for the gradients. By following these
rules, you get a method adaptive to the local geometry, with convergence
guarantees depending only on the smoothness in a neighborhood of a solution.
Given that the problem is convex, our method converges even if the global
smoothness constant is infinity. As an illustration, it can minimize arbitrary
continuously twice-differentiable convex function. We examine its performance
on a range of convex and nonconvex problems, including logistic regression and
matrix factorization.

本文提供一个简明的证明，只需遵循两个规则即可自动化梯度下降：1）不要过快增加步长，2）不要超出局部曲率；通过遵循这些规则，可以得到对局部几何条件自适应的方法，收敛保证只取决于解的附近的平滑度，因此收敛于任何凸问题中，包括可以最小化任意连续两次可微的凸函数的问题，本文将探讨该方法在一系列凸和非凸问题上的性能。

自适应梯度下降（无需下降）

Adaptive Gradient Descent without Descent

We define a second-order neural network stochastic gradient training
algorithm whose block-diagonal structure effectively amounts to normalizing the
unit activations. Investigating why this algorithm lacks in robustness then
reveals two interesting insights. The first insight suggests a new way to scale
the stepsizes, clarifying popular algorithms such as RMSProp as well as old
neural network tricks such as fanin stepsize scaling. The second insight
stresses the practical importance of dealing with fast changes of the curvature
of the cost.

定义了一种二阶神经网络随机梯度训练算法，其块对角结构有效地实现了单元激活的归一化，并研究了该算法在稳健性方面的不足之处，揭示了步长缩放的新方式以及处理成本曲率快速变化的重要性。