We present practical Levenberg-Marquardt variants of Gauss-Newton and natural
gradient methods for solving non-convex optimization problems that arise in
training deep neural networks involving enormous numbers of variables and huge
data sets. Our methods use subsampled Gauss-Newton or Fisher information
matrices and either subsampled gradient estimates (fully stochastic) or full
gradients (semi-stochastic), which, in the latter case, we prove convergent to
a stationary point. By using the Sherman-Morrison-Woodbury formula with
automatic differentiation (backpropagation) we show how our methods can be
implemented to perform efficiently. Finally, numerical results are presented to
demonstrate the effectiveness of our proposed methods.

使用 Levenberg-Marquardt 的 Gauss-Newton 和自然梯度方法，解决深度神经网络大量变量和海量数据集所产生的非凸优化问题，证明方法能够有效实现并提出数值结果。

高效的子采样 Gauss-Newton 和自然梯度方法用于训练神经网络

Efficient Subsampled Gauss-Newton and Natural Gradient Methods for  Training Neural Networks

We propose a continuous optimization method for solving dense 3D scene flow
problems from stereo imagery. As in recent work, we represent the dynamic 3D
scene as a collection of rigidly moving planar segments. The scene flow problem
then becomes the joint estimation of pixel-to-segment assignment, 3D position,
normal vector and rigid motion parameters for each segment, leading to a
complex and expensive discrete-continuous optimization problem. In contrast, we
propose a purely continuous formulation which can be solved more efficiently.
Using a fine superpixel segmentation that is fixed a-priori, we propose a
factor graph formulation that decomposes the problem into photometric,
geometric, and smoothing constraints. We initialize the solution with a novel,
high-quality initialization method, then independently refine the geometry and
motion of the scene, and finally perform a global non-linear refinement using
Levenberg-Marquardt. We evaluate our method in the challenging KITTI Scene Flow
benchmark, ranking in third position, while being 3 to 30 times faster than the
top competitors.

本文提出了一种用于从立体影像中求解密集的三维场景流问题的连续优化方法，其中将动态三维场景表示为一系列刚性运动的平面段，使用预先确定的精细超像素分割，将问题分解为光度学、几何和平滑约束，并使用 Levenberg-Marquardt 进行全局非线性优化。