Gauss-Newton methods and their stochastic version have been widely used in
machine learning and signal processing. Their nonsmooth counterparts, modified
Gauss-Newton or prox-linear algorithms, can lead to contrasting outcomes when
compared to gradient descent in large-scale statistical settings. We explore
the contrasting performance of these two classes of algorithms in theory on a
stylized statistical example, and experimentally on learning problems including
structured prediction. In theory, we delineate the regime where the quadratic
convergence of the modified Gauss-Newton method is active under statistical
noise. In the experiments, we underline the versatility of stochastic
(sub)-gradient descent to minimize nonsmooth composite objectives.

本文研究了在大规模统计学环境中用于机器学习和信号处理的 Gauss-Newton 方法及其随机版本，以及它们的非光滑对应物 prox-linear 算法。该文在一个简化的统计学例子和结构化预测学习问题上，对这两类算法的对比表现进行了理论和实验研究，着重研究了在统计噪声下 modified Gauss-Newton 方法二次收敛的适用范围，并强调了随机梯度下降优化非光滑复合目标函数的多用途性。