A recent line of work has established intriguing connections between the
generalization/compression properties of a deep neural network (DNN) model and
the so-called layer weights' stable ranks. Intuitively, the latter are
indicators of the effective number of parameters in the net. In this work, we
address some natural questions regarding the space of DNNs conditioned on the
layers' stable rank, where we study feed-forward dynamics, initialization,
training and expressivity. To this end, we first propose a random DNN model
with a new sampling scheme based on stable rank. Then, we show how feed-forward
maps are affected by the constraint and how training evolves in the
overparametrized regime (via Neural Tangent Kernels). Our results imply that
stable ranks appear layerwise essentially as linear factors whose effect
accumulates exponentially depthwise. Moreover, we provide empirical analysis
suggesting that stable rank initialization alone can lead to convergence speed
ups.

本研究探讨了深度神经网络模型稳定秩的空间，分析了前向动力学、初始化、训练和表达性等方面，并通过实证分析表明，仅稳定秩初始化即可加速收敛速度。