We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks. Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task, despite the presence of other predictive features that fail to be discovered. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks. Using tools from Dynamical Systems theory, we identify simple properties of learning dynamics during gradient descent that lead to this imbalance, and prove that such a situation can be expected given certain statistical structure in training data. Based on our proposed formalism, we develop guarantees for a novel regularization method aimed at decoupling feature learning dynamics, improving accuracy and robustness in cases hindered by gradient starvation. We illustrate our findings with simple and real-world out-of-distribution (OOD) generalization experiments.

本文探讨超参数神经网络学习中的梯度下降现象，发现其在最小化交叉熵损失时可能只捕获部分特征，而导致特征的不平衡。作者提出了一种理论解释，并使用动力系统理论中的工具来证明给定训练数据的某些统计结构时可以预期这种情况。此外，作者还提出了一种新的正则化方法来解决梯度饱和问题，并且在实验中得到了验证。

梯度饱和：神经网络的学习偏好