For the convolutional neural network (CNN) used for pattern classification,
the training loss function is usually applied to the final output of the
network, except for some regularization constraints on the network parameters.
However, with the increasing of the number of network layers, the influence of
the loss function on the network front layers gradually decreases, and the
network parameters tend to fall into local optimization. At the same time, it
is found that the trained network has significant information redundancy at all
stages of features, which reduces the effectiveness of feature mapping at all
stages and is not conducive to the change of the subsequent parameters of the
network in the direction of optimality. Therefore, it is possible to obtain a
more optimized solution of the network and further improve the classification
accuracy of the network by designing a loss function for restraining the front
stage features and eliminating the information redundancy of the front stage
features .For CNN, this article proposes a multi-stage feature decorrelation
loss (MFD Loss), which refines effective features and eliminates information
redundancy by constraining the correlation of features at all stages.
Considering that there are many layers in CNN, through experimental comparison
and analysis, MFD Loss acts on multiple front layers of CNN, constrains the
output features of each layer and each channel, and performs supervision
training jointly with classification loss function during network training.
Compared with the single Softmax Loss supervised learning, the experiments on
several commonly used datasets on several typical CNNs prove that the
classification performance of Softmax Loss+MFD Loss is significantly better.
Meanwhile, the comparison experiments before and after the combination of MFD
Loss and some other typical loss functions verify its good universality.

为了优化卷积神经网络的训练结果，本文提出了一种多阶段特征去相关损失函数（MFD Loss），通过约束各个阶段的特征相关性，优化特征映射，进一步提高分类准确性。与单一 Softmax Loss 相比，使用 Softmax Loss+MFD Loss 的分类性能明显更好，并且实验证明 MFD Loss 与其他典型损失函数的组合效果也很好。

改进 CNN 分类性能的多阶段特征去相关约束

Multi-stage feature decorrelation constraints for improving CNN  classification performance

Many important computer vision tasks are naturally formulated to have a
non-differentiable objective. Therefore, the standard, dominant training
procedure of a neural network is not applicable since back-propagation requires
the gradients of the objective with respect to the output of the model. Most
deep learning methods side-step the problem sub-optimally by using a proxy loss
for training, which was originally designed for another task and is not
tailored to the specifics of the objective. The proxy loss functions may or may
not align well with the original non-differentiable objective. An appropriate
proxy has to be designed for a novel task, which may not be feasible for a
non-specialist. This thesis makes four main contributions toward bridging the
gap between the non-differentiable objective and the training loss function.
Throughout the thesis, we refer to a loss function as a surrogate loss if it is
a differentiable approximation of the non-differentiable objective. Note that
we use the terms objective and evaluation metric interchangeably.
The contributions of this thesis make the training of neural networks more
scalable -- to new tasks in a nearly labor-free manner when the evaluation
metric is decomposable, which will help researchers with novel tasks. For
non-decomposable evaluation metrics, the differentiable components developed
for the recall@k surrogate, such as sorting and counting, can also be used for
creating new surrogates.

本论文探讨了计算机视觉领域中非可微目标函数的训练问题，提出了使用可微的代理函数进行训练的解决方案，使得神经网络训练更易扩展到新型任务，包括可分解和不可分解的评估指标。

神经网络训练与非可微目标函数

Neural Network Training and Non-Differentiable Objective Functions

As shown in recent research, deep neural networks can perfectly fit randomly
labeled data, but with very poor accuracy on held out data. This phenomenon
indicates that loss functions such as cross-entropy are not a reliable
indicator of generalization. This leads to the crucial question of how
generalization gap should be predicted from the training data and network
parameters. In this paper, we propose such a measure, and conduct extensive
empirical studies on how well it can predict the generalization gap. Our
measure is based on the concept of margin distribution, which are the distances
of training points to the decision boundary. We find that it is necessary to
use margin distributions at multiple layers of a deep network. On the CIFAR-10
and the CIFAR-100 datasets, our proposed measure correlates very strongly with
the generalization gap. In addition, we find the following other factors to be
of importance: normalizing margin values for scale independence, using
characterizations of margin distribution rather than just the margin (closest
distance to decision boundary), and working in log space instead of linear
space (effectively using a product of margins rather than a sum). Our measure
can be easily applied to feedforward deep networks with any architecture and
may point towards new training loss functions that could enable better
generalization.

该论文研究发现，交叉熵等损失函数不能很好地预测深度神经网络的泛化能力，作者提出了一种基于边缘分布的测量方法，它可以被应用在任何架构的前馈深度网络上，并指出这个方法可能会提示新的训练损失函数的设计来实现更好的泛化。