When humans learn a new concept, they might ignore examples that they cannot make sense of at first, and only later focus on such examples, when they are more useful for learning. We propose incorporating this idea of tunable sensitivity for hard examples in neural network learning, using a new generalization of the cross-entropy gradient step, which can be used in place of the gradient in any gradient-based training method. The generalized gradient is parameterized by a value that controls the sensitivity of the training process to harder training examples. We tested our method on several benchmark datasets. We propose, and corroborate in our experiments, that the optimal level of sensitivity to hard example is positively correlated with the depth of the network. Moreover, the test prediction error obtained by our method is generally lower than that of the vanilla cross-entropy gradient learner. We therefore conclude that tunable sensitivity can be helpful for neural network learning.

本研究提出了一种改进的交叉熵梯度步骤，该步骤可以用于基于梯度的任何训练方法，并通过控制训练过程对较难的训练样本的敏感度来使用可调感度以帮助神经网络学习，该方法在多个基准数据集上测试后证实了这一点。

神经网络训练中对大误差的可调灵敏度