While a lot of progress has been made in recent years, the dynamics of
learning in deep nonlinear neural networks remain to this day largely
misunderstood. In this work, we study the case of binary classification and
prove various properties of learning in such networks under strong assumptions
such as linear separability of the data. Extending existing results from the
linear case, we confirm empirical observations by proving that the
classification error also follows a sigmoidal shape in nonlinear architectures.
We show that given proper initialization, learning expounds parallel
independent modes and that certain regions of parameter space might lead to
failed training. We also demonstrate that input norm and features' frequency in
the dataset lead to distinct convergence speeds which might shed some light on
the generalization capabilities of deep neural networks. We provide a
comparison between the dynamics of learning with cross-entropy and hinge
losses, which could prove useful to understand recent progress in the training
of generative adversarial networks. Finally, we identify a phenomenon that we
baptize gradient starvation where the most frequent features in a dataset
prevent the learning of other less frequent but equally informative features.

研究深度神经网络的学习动态，主要关注于二元分类问题。我们证明了网络学习的各种性质，并且在非线性架构下，分类误差也呈现出 sigmoid 形状，证实了经验观察。我们指出了梯度饱和现象和特征频率对模型收敛速度的影响，并探讨了交叉熵和 hinge 损失对生成对抗网络训练的差异。最后，我们提出了梯度饥饿现象并进行了研究。