We analyze algorithms for approximating a function $f(x) = \Phi x$ mapping
$\Re^d$ to $\Re^d$ using deep linear neural networks, i.e. that learn a
function $h$ parameterized by matrices $\Theta_1,...,\Theta_L$ and defined by
$h(x) = \Theta_L \Theta_{L-1} ... \Theta_1 x$. We focus on algorithms that
learn through gradient descent on the population quadratic loss in the case
that the distribution over the inputs is isotropic.
We provide polynomial bounds on the number of iterations for gradient descent
to approximate the least squares matrix $\Phi$, in the case where the initial
hypothesis $\Theta_1 = ... = \Theta_L = I$ has excess loss bounded by a small
enough constant. On the other hand, we show that gradient descent fails to
converge for $\Phi$ whose distance from the identity is a larger constant, and
we show that some forms of regularization toward the identity in each layer do
not help.
If $\Phi$ is symmetric positive definite, we show that an algorithm that
initializes $\Theta_i = I$ learns an $\epsilon$-approximation of $f$ using a
number of updates polynomial in $L$, the condition number of $\Phi$, and
$\log(d/\epsilon)$. In contrast, we show that if the least squares matrix
$\Phi$ is symmetric and has a negative eigenvalue, then all members of a class
of algorithms that perform gradient descent with identity initialization, and
optionally regularize toward the identity in each layer, fail to converge.
We analyze an algorithm for the case that $\Phi$ satisfies $u^{\top} \Phi u >
0$ for all $u$, but may not be symmetric. This algorithm uses two regularizers:
one that maintains the invariant $u^{\top} \Theta_L \Theta_{L-1} ... \Theta_1 u
> 0$ for all $u$, and another that "balances" $\Theta_1, ..., \Theta_L$ so that
they have the same singular values.

本文介绍了使用梯度下降算法在深度线性神经网络中学习的近似函数的算法，并分析了在输入分布为各向同性时，使用多项式约束的梯度下降算法能多快地逼近最小二乘矩阵，以及针对不同情况下的正则化方法如何对收敛时效产生影响。此外，文章还探讨了对称正定矩阵条件下的算法，以及对于某些不对称矩阵的处理方式。

深度残差网络通过身份初始化高效学习正定线性变换的梯度下降

Gradient descent with identity initialization efficiently learns  positive definite linear transformations by deep residual networks

Symmetric Positive Definite (SPD) matrices have become popular to encode
image information. Accounting for the geometry of the Riemannian manifold of
SPD matrices has proven key to the success of many algorithms. However, most
existing methods only approximate the true shape of the manifold locally by its
tangent plane. In this paper, inspired by kernel methods, we propose to map SPD
matrices to a high dimensional Hilbert space where Euclidean geometry applies.
To encode the geometry of the manifold in the mapping, we introduce a family of
provably positive definite kernels on the Riemannian manifold of SPD matrices.
These kernels are derived from the Gaussian ker- nel, but exploit different
metrics on the manifold. This lets us extend kernel-based algorithms developed
for Euclidean spaces, such as SVM and kernel PCA, to the Riemannian manifold of
SPD matrices. We demonstrate the benefits of our approach on the problems of
pedestrian detection, ob- ject categorization, texture analysis, 2D motion
segmentation and Diffusion Tensor Imaging (DTI) segmentation.

该文提出了一种针对 Riemann 流形的 SPD 矩阵进行高维映射的方法，使用一组可证明的正定核函数来扩展基于核方法的算法，进而在人行检测、物体分类、纹理分析、2D 运动分割以及扩散张量成像分割等问题上取得了良好的效果。