In deep learning classifiers, the cost function usually takes the form of a
combination of SoftMax and CrossEntropy functions. The SoftMax unit transforms
the scores predicted by the model network into assessments of the degree
(probabilities) of an object's membership to a given class. On the other hand,
CrossEntropy measures the divergence of this prediction from the distribution
of target scores. This work introduces the ISBE functionality, justifying the
thesis about the redundancy of cross entropy computation in deep learning of
classifiers. Not only can we omit the calculation of entropy, but also, during
back-propagation, there is no need to direct the error to the normalization
unit for its backward transformation. Instead, the error is sent directly to
the model's network. Using examples of perceptron and convolutional networks as
classifiers of images from the MNIST collection, it is observed for ISBE that
results are not degraded with SoftMax only, but also with other activation
functions such as Sigmoid, Tanh, or their hard variants HardSigmoid and
HardTanh. Moreover, up to three percent of time is saved within the total time
of forward and backward stages. The article is addressed mainly to programmers
and students interested in deep model learning. For example, it illustrates in
code snippets possible ways to implement ISBE units, but also formally proves
that the softmax trick only applies to the class of softmax functions with
relocations.

该论文介绍了 ISBE 功能，证明了在深度学习分类器中计算熵是多余的，同时指出误差可以直接发送到模型网络，且使用 SoftMax 以及其他激活函数时结果不会下降，还可以节省总时间的 3％。