Inspired by Gauss-Newton-like methods, we study the benefit of leveraging the structure of deep learning objectives, namely, the composition of a convex loss function and of a nonlinear network, in order to derive better direction oracles than stochastic gradients, based on the idea of partial linearization. In a departure from previous works, we propose to compute such direction oracles via their dual formulation, leading to both computational benefits and new insights. We demonstrate that the resulting oracles define descent directions that can be used as a drop-in replacement for stochastic gradients, in existing optimization algorithms. We empirically study the advantage of using the dual formulation as well as the computational trade-offs involved in the computation of such oracles.

通过利用深度学习目标的结构，即凸损失函数和非线性网络的组合，我们研究了借鉴类高斯-牛顿方法的好处，以便通过部分线性化的思想，导出比随机梯度更好的方向神经元，通过其对偶式给出这些方向神经元的计算好处并带来新的见解。我们证明了所得到的方向神经元定义了可以作为随机梯度的替代品的下降方向，在现有优化算法中可以直接使用。我们经验性地研究了使用对偶式的优势以及计算此类方向神经元所涉及的计算权衡。

深度学习的双高斯牛顿方向