While the superior performance of second-order optimization methods such as
Newton's method is well known, they are hardly used in practice for deep
learning because neither assembling the Hessian matrix nor calculating its
inverse is feasible for large-scale problems. Existing second-order methods
resort to various diagonal or low-rank approximations of the Hessian, which
often fail to capture necessary curvature information to generate a substantial
improvement. On the other hand, when training becomes batch-based (i.e.,
stochastic), noisy second-order information easily contaminates the training
procedure unless expensive safeguard is employed. In this paper, we adopt a
numerical algorithm for second-order neural network training. We tackle the
practical obstacle of Hessian calculation by using the complex-step finite
difference (CSFD) -- a numerical procedure adding an imaginary perturbation to
the function for derivative computation. CSFD is highly robust, efficient, and
accurate (as accurate as the analytic result). This method allows us to
literally apply any known second-order optimization methods for deep learning
training. Based on it, we design an effective Newton Krylov procedure. The key
mechanism is to terminate the stochastic Krylov iteration as soon as a
disturbing direction is found so that unnecessary computation can be avoided.
During the optimization, we monitor the approximation error in the Taylor
expansion to adjust the step size. This strategy combines advantages of line
search and trust region methods making our method preserves good local and
global convergency at the same time. We have tested our methods in various deep
learning tasks. The experiments show that our method outperforms exiting
methods, and it often converges one-order faster. We believe our method will
inspire a wide-range of new algorithms for deep learning and numerical
optimization.

本文提出了一个基于二阶数值优化的深度学习算法。该算法使用复合数算法流的有限差分（CDSD）计算海森矩阵，通过监控泰勒级数的逼近误差，调整步长大小，实现了优化的同时保存良好的局部和全局收敛性，在深度学习任务中表现优异。