We identify and formalize a fundamental gradient descent phenomenon resulting
in a learning proclivity in over-parameterized neural networks. Gradient
Starvation arises when cross-entropy loss is minimized by capturing only a
subset of features relevant for the task, despite the presence of other
predictive features that fail to be discovered. This work provides a
theoretical explanation for the emergence of such feature imbalance in neural
networks. Using tools from Dynamical Systems theory, we identify simple
properties of learning dynamics during gradient descent that lead to this
imbalance, and prove that such a situation can be expected given certain
statistical structure in training data. Based on our proposed formalism, we
develop guarantees for a novel regularization method aimed at decoupling
feature learning dynamics, improving accuracy and robustness in cases hindered
by gradient starvation. We illustrate our findings with simple and real-world
out-of-distribution (OOD) generalization experiments.

本文探讨超参数神经网络学习中的梯度下降现象，发现其在最小化交叉熵损失时可能只捕获部分特征，而导致特征的不平衡。作者提出了一种理论解释，并使用动力系统理论中的工具来证明给定训练数据的某些统计结构时可以预期这种情况。此外，作者还提出了一种新的正则化方法来解决梯度饱和问题，并且在实验中得到了验证。

梯度饱和：神经网络的学习偏好

Gradient Starvation: A Learning Proclivity in Neural Networks

While a lot of progress has been made in recent years, the dynamics of
learning in deep nonlinear neural networks remain to this day largely
misunderstood. In this work, we study the case of binary classification and
prove various properties of learning in such networks under strong assumptions
such as linear separability of the data. Extending existing results from the
linear case, we confirm empirical observations by proving that the
classification error also follows a sigmoidal shape in nonlinear architectures.
We show that given proper initialization, learning expounds parallel
independent modes and that certain regions of parameter space might lead to
failed training. We also demonstrate that input norm and features' frequency in
the dataset lead to distinct convergence speeds which might shed some light on
the generalization capabilities of deep neural networks. We provide a
comparison between the dynamics of learning with cross-entropy and hinge
losses, which could prove useful to understand recent progress in the training
of generative adversarial networks. Finally, we identify a phenomenon that we
baptize gradient starvation where the most frequent features in a dataset
prevent the learning of other less frequent but equally informative features.

研究深度神经网络的学习动态，主要关注于二元分类问题。我们证明了网络学习的各种性质，并且在非线性架构下，分类误差也呈现出 sigmoid 形状，证实了经验观察。我们指出了梯度饱和现象和特征频率对模型收敛速度的影响，并探讨了交叉熵和 hinge 损失对生成对抗网络训练的差异。最后，我们提出了梯度饥饿现象并进行了研究。