Self-training is a powerful approach to deep learning. The key process is to
find a pseudo-label for modeling. However, previous self-training algorithms
suffer from the over-confidence issue brought by the hard labels, even some
confidence-related regularizers cannot comprehensively catch the uncertainty.
Therefore, we propose a new self-training framework to combine uncertainty
information of both model and dataset. Specifically, we propose to use
Expectation-Maximization (EM) to smooth the labels and comprehensively estimate
the uncertainty information. We further design a basis extraction network to
estimate the initial basis from the dataset. The obtained basis with
uncertainty can be filtered based on uncertainty information. It can then be
transformed into the real hard label to iteratively update the model and basis
in the retraining process. Experiments on image classification and semantic
segmentation show the advantages of our methods among confidence-aware
self-training algorithms with 1-3 percentage improvement on different datasets.

我们提出了一种新的自训练框架，用于结合模型和数据集的不确定性信息，通过使用期望最大化 (EM) 平滑标签和全面估计不确定性信息，并设计一种基础提取网络从数据集中估计初始基础，然后通过不确定性信息将其转换为真实硬标签，以在重新训练过程中迭代更新模型和基础。在图像分类和语义分割的实验证明了我们的方法在可信自训练算法中具有优势，在不同数据集上可以获得 1-3 个百分点的改进。

基于期望最大化变换的不确定性感知自训练

Uncertainty-aware self-training with expectation maximization basis  transformation

How to develop slim and accurate deep neural networks has become crucial for
real- world applications, especially for those employed in embedded systems.
Though previous work along this research line has shown some promising results,
most existing methods either fail to significantly compress a well-trained deep
network or require a heavy retraining process for the pruned deep network to
re-boost its prediction performance. In this paper, we propose a new layer-wise
pruning method for deep neural networks. In our proposed method, parameters of
each individual layer are pruned independently based on second order
derivatives of a layer-wise error function with respect to the corresponding
parameters. We prove that the final prediction performance drop after pruning
is bounded by a linear combination of the reconstructed errors caused at each
layer. Therefore, there is a guarantee that one only needs to perform a light
retraining process on the pruned network to resume its original prediction
performance. We conduct extensive experiments on benchmark datasets to
demonstrate the effectiveness of our pruning method compared with several
state-of-the-art baseline methods.

本文提出了一种基于层次的剪枝方法对深度神经网络进行压缩，通过独立剪枝每个层次的参数可以保证压缩后的网络只需要进行轻量级的重新训练即可恢复预测能力，并在基准数据集上进行了广泛实验以证明其有效性。