We study regularized deep neural networks (DNNs) and introduce a convex
analytic framework to characterize the structure of the hidden layers. We show
that a set of optimal hidden layer weights for a norm regularized DNN training
problem can be explicitly found as the extreme points of a convex set. For the
special case of deep linear networks, we prove that each optimal weight matrix
aligns with the previous layers via duality. More importantly, we apply the
same characterization to deep ReLU networks with whitened data and prove the
same weight alignment holds. As a corollary, we also prove that norm
regularized deep ReLU networks yield spline interpolation for one-dimensional
datasets which was previously known only for two-layer networks. Furthermore,
we provide closed-form solutions for the optimal layer weights when data is
rank-one or whitened. The same analysis also applies to architectures with
batch normalization even for arbitrary data. Therefore, we obtain a complete
explanation for a recent empirical observation termed Neural Collapse where
class means collapse to the vertices of a simplex equiangular tight frame.

本文研究正则化深度神经网络及其隐层结构，通过凸分析框架构建问题的最优隐层权重，证明 For 深度 ReLU 网络，权重矩阵与之前的层通过对偶对齐，并给出了数据为基态或白话时的权重的解析解。同时，该研究也可以甚至适用于具有批归一化架构的深度神经网络，并给出了 “神经坍塌” 现象的完整解释。

通过凸对偶揭示深度神经网络的结构

Revealing the Structure of Deep Neural Networks via Convex Duality

We analyze speed of convergence to global optimum for gradient descent
training a deep linear neural network (parameterized as $x \mapsto W_N W_{N-1}
\cdots W_1 x$) by minimizing the $\ell_2$ loss over whitened data. Convergence
at a linear rate is guaranteed when the following hold: (i) dimensions of
hidden layers are at least the minimum of the input and output dimensions; (ii)
weight matrices at initialization are approximately balanced; and (iii) the
initial loss is smaller than the loss of any rank-deficient solution. The
assumptions on initialization (conditions (ii) and (iii)) are necessary, in the
sense that violating any one of them may lead to convergence failure. Moreover,
in the important case of output dimension 1, i.e. scalar regression, they are
met, and thus convergence to global optimum holds, with constant probability
under a random initialization scheme. Our results significantly extend previous
analyses, e.g., of deep linear residual networks (Bartlett et al., 2018).

本文研究在白化数据上，通过梯度下降来训练深度线性神经网络收敛到全局最优点的速度。当隐藏层数的维度不小于输入输出维度的最小值，并且初始化的权重矩阵大致平衡且初始损失小于任何秩缺失解时，可保证线性收敛。此外，在输出维度为 1 的情况下，即标量回归，这些条件是满足的，并且在随机初始化方案下具有恒定的概率达到全局最优解。