Unsupervised domain adaptation (UDA) aims to adapt existing models of the
source domain to a new target domain with only unlabeled data. Most existing
methods suffer from noticeable negative transfer resulting from either the
error-prone discriminator network or the unreasonable teacher model. Besides,
the local regional consistency in UDA has been largely neglected, and only
extracting the global-level pattern information is not powerful enough for
feature alignment due to the abuse use of contexts. To this end, we propose an
uncertainty-aware consistency regularization method for cross-domain semantic
segmentation. Firstly, we introduce an uncertainty-guided consistency loss with
a dynamic weighting scheme by exploiting the latent uncertainty information of
the target samples. As such, more meaningful and reliable knowledge from the
teacher model can be transferred to the student model. We further reveal the
reason why the current consistency regularization is often unstable in
minimizing the domain discrepancy. Besides, we design a ClassDrop mask
generation algorithm to produce strong class-wise perturbations. Guided by this
mask, we propose a ClassOut strategy to realize effective regional consistency
in a fine-grained manner. Experiments demonstrate that our method outperforms
the state-of-the-art methods on four domain adaptation benchmarks, i.e., GTAV
$\rightarrow $ Cityscapes and SYNTHIA $\rightarrow $ Cityscapes, Virtual KITTI
$\rightarrow$ KITTI and Cityscapes $\rightarrow$ KITTI.

提出了一种基于不确定性感知和地区一致性的跨域语义分割一致性正则化方法，通过利用目标样本的潜在不确定性信息，引入了动态加权的一致性损失，使得从教师模型传递到学生模型的知识更具意义和可靠性。同时，提出了一种 ClassDrop 掩蔽生成算法和 ClassOut 策略，实现了地区一致性的有效建模。实验表明，该方法在四个跨域自适应基准上优于现有方法。

跨域语义分割的不确定性感知一致性正则化

Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic  Segmentation

Knowledge distillation (KD) consists of transferring knowledge from one
machine learning model (the teacher}) to another (the student). Commonly, the
teacher is a high-capacity model with formidable performance, while the student
is more compact. By transferring knowledge, one hopes to benefit from the
student's compactness. %we desire a compact model with performance close to the
teacher's. We study KD from a new perspective: rather than compressing models,
we train students parameterized identically to their teachers. Surprisingly,
these {Born-Again Networks (BANs), outperform their teachers significantly,
both on computer vision and language modeling tasks. Our experiments with BANs
based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10
(3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional
experiments explore two distillation objectives: (i) Confidence-Weighted by
Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP).
Both methods elucidate the essential components of KD, demonstrating a role of
the teacher outputs on both predicted and non-predicted classes. We present
experiments with students of various capacities, focusing on the under-explored
case where students overpower teachers. Our experiments show significant
advantages from transferring knowledge between DenseNets and ResNets in either
direction.

本文研究知识迁移领域的另一种方法：Born-Again Networks (BANs)，将学生的参数与老师的相同， 来达到比老师更好的性能表现。在计算机视觉和语言建模任务中， BANs 表现出甚至比老师高出许多的性能得分，并且在各种规模的学生中，将知识从 DenseNets 到 ResNets 和反向传输，都显示出明显的优势

再生神经网络

Born Again Neural Networks

Some machine learning applications involve training data that is sensitive,
such as the medical histories of patients in a clinical trial. A model may
inadvertently and implicitly store some of its training data; careful analysis
of the model may therefore reveal sensitive information.
To address this problem, we demonstrate a generally applicable approach to
providing strong privacy guarantees for training data: Private Aggregation of
Teacher Ensembles (PATE). The approach combines, in a black-box fashion,
multiple models trained with disjoint datasets, such as records from different
subsets of users. Because they rely directly on sensitive data, these models
are not published, but instead used as "teachers" for a "student" model. The
student learns to predict an output chosen by noisy voting among all of the
teachers, and cannot directly access an individual teacher or the underlying
data or parameters. The student's privacy properties can be understood both
intuitively (since no single teacher and thus no single dataset dictates the
student's training) and formally, in terms of differential privacy. These
properties hold even if an adversary can not only query the student but also
inspect its internal workings.
Compared with previous work, the approach imposes only weak assumptions on
how teachers are trained: it applies to any model, including non-convex models
like DNNs. We achieve state-of-the-art privacy/utility trade-offs on MNIST and
SVHN thanks to an improved privacy analysis and semi-supervised learning.

本研究提出了一种称为私有教师集成（PATE）的方法，通过多个使用不同数据集训练的模型进行加密学习，从而实现对模型训练数据的隐私保护，达到最新的隐私 / 效用平衡，且适用于包括非凸模型在内的任何模型。