Generalization and optimization guarantees on the population loss in machine
learning often rely on uniform convergence based analysis, typically based on
the Rademacher complexity of the predictors. The rich representation power of
modern models has led to concerns about this approach. In this paper, we
present generalization and optimization guarantees in terms of the complexity
of the gradients, as measured by the Loss Gradient Gaussian Width (LGGW).
First, we introduce generalization guarantees directly in terms of the LGGW
under a flexible gradient domination condition, which we demonstrate to hold
empirically for deep models. Second, we show that sample reuse in finite sum
(stochastic) optimization does not make the empirical gradient deviate from the
population gradient as long as the LGGW is small. Third, focusing on deep
networks, we present results showing how to bound their LGGW under mild
assumptions. In particular, we show that their LGGW can be bounded (a) by the
$L_2$-norm of the loss Hessian eigenvalues, which has been empirically shown to
be $\tilde{O}(1)$ for commonly used deep models; and (b) in terms of the
Gaussian width of the featurizer, i.e., the output of the last-but-one layer.
To our knowledge, our generalization and optimization guarantees in terms of
LGGW are the first results of its kind, avoid the pitfalls of predictor
Rademacher complexity based analysis, and hold considerable promise towards
quantitatively tight bounds for deep models.

通过 Loss Gradient Gaussian Width (LGGW) 的复杂度度量，我们提出一种直接基于 LGGW 的泛化保证方法，在梯度支配条件下实现，其在深度模型中进行实证研究并证明其有效性。同时，我们展示了有限和优化的样本重用并不会导致经验梯度偏离总体梯度，只要 LGGW 较小。这些基于 LGGW 的优化和泛化保证是首次提出的，避免了基于预测器 Rademacher 复杂性的分析缺陷，并对深度模型的定量界限提供了重要前景。

基于损失梯度高斯宽度的泛化和优化保证

Loss Gradient Gaussian Width based Generalization and Optimization  Guarantees

Two common approaches in low-rank optimization problems are either working
directly with a rank constraint on the matrix variable, or optimizing over a
low-rank factorization so that the rank constraint is implicitly ensured. In
this paper, we study the natural connection between the rank-constrained and
factorized approaches. We show that all second-order stationary points of the
factorized objective function correspond to fixed points of projected gradient
descent run on the original problem (where the projection step enforces the
rank constraint). This result allows us to unify many existing optimization
guarantees that have been proved specifically in either the rank-constrained or
the factorized setting, and leads to new results for certain settings of the
problem. We demonstrate application of our results to several concrete low-rank
optimization problems arising in matrix inverse problems.

本文研究了低秩优化问题中矩阵变量秩约束和低秩分解优化方法的自然联系，证明了低秩分解目标函数的所有二阶稳定点对应于原问题上投影梯度下降算法的不动点，并统一了现有的优化保证，为某些情况下特定的问题提供了新结果。最后，将结果应用于矩阵逆问题。

排名限制与低秩分解的临界点之间的等价性

An equivalence between critical points for rank constraints versus  low-rank factorizations

This work aims to provide understandings on the remarkable success of deep
convolutional neural networks (CNNs) by theoretically analyzing their
generalization performance and establishing optimization guarantees for
gradient descent based training algorithms. Specifically, for a CNN model
consisting of $l$ convolutional layers and one fully connected layer, we prove
that its generalization error is bounded by
$\mathcal{O}(\sqrt{\dt\widetilde{\varrho}/n})$ where $\theta$ denotes freedom
degree of the network parameters and
$\widetilde{\varrho}=\mathcal{O}(\log(\prod_{i=1}^{l}\rwi{i}
(\ki{i}-\si{i}+1)/p)+\log(\rf))$ encapsulates architecture parameters including
the kernel size $\ki{i}$, stride $\si{i}$, pooling size $p$ and parameter
magnitude $\rwi{i}$. To our best knowledge, this is the first generalization
bound that only depends on $\mathcal{O}(\log(\prod_{i=1}^{l+1}\rwi{i}))$,
tighter than existing ones that all involve an exponential term like
$\mathcal{O}(\prod_{i=1}^{l+1}\rwi{i})$. Besides, we prove that for an
arbitrary gradient descent algorithm, the computed approximate stationary point
by minimizing empirical risk is also an approximate stationary point to the
population risk. This well explains why gradient descent training algorithms
usually perform sufficiently well in practice. Furthermore, we prove the
one-to-one correspondence and convergence guarantees for the non-degenerate
stationary points between the empirical and population risks. It implies that
the computed local minimum for the empirical risk is also close to a local
minimum for the population risk, thus ensuring the good generalization
performance of CNNs.

本文理论分析卷积神经网络（CNN）的泛化性能、梯度下降训练算法的优化保证，证明了 CNN 的泛化误差由自由度和体系结构参数决定，并证明了梯度下降算法的近似稳定点是人口风险的近似稳定点，保证了 CNN 的良好泛化性能。